Gergely Nagy <algernon@balabit.hu> writes:

"Lucas, Sascha" <Sascha.Lucas@gisa.de> writes:

...

Hi,

Mongodb 2.2 has a new feature called TTL collections (expire data by setting a TTL). An analogy to the classic logrotate. One constraint is, that you index a field of date BSON type[1]. As far as I know, atm syslog-ng can only store strings? Or how can it be done to store one field as date?

Right now, only with an external program. With syslog-ng 3.3, it will stay that way, for 3.4, I have a half-baked patch that adds a date() option to the mongodb destination.

I will backport that patch to 3.3 aswell, and it will be available as a separate patch, but not as part of the official release.

If all goes well, this will be done by this weekend. Hopefully this'll do until syslog-ng learns how to transfer type information too (which is something I want to do for 3.4 too, but may not have time for it).

This took a little more than the next weekend (I really should stop trying to estimate, it hardly ever works), but I just sent a patch to the list for 3.4 that allows one to do this: mongodb(datetime("some.field.name") value-pairs(pair("some.field.name", "$UNIXTIME"))); That is, whatever you put in as the value of datetime(), will be searched for in the list of value pairs, and if a field is found, its value will be converted to proper datetime-typed thing. The code assumes that the value is a unix timestamp, and will not insert the value if it fails the conversion. I have not backported it to 3.3 yet, as the code there is considerably different, but a similar approach can be taken there too - and it will be even simpler to achieve the same thing. Nevertheless, I'll do the backport in a couple of minutes, the results will be available on my feature/3.3/mongodb/datetime branch shortly after. -- |8]

Hi Gergely, Am Donnerstag, den 08.11.2012, 11:21 +0000 schrieb Gergely Nagy:

trying to estimate, it hardly ever works), but I just sent a patch to the list for 3.4 that allows one to do this:

mongodb(datetime("some.field.name") value-pairs(pair("some.field.name", "$UNIXTIME")));

was the patch you mention above accepted for syslog-ng-OSE-3.4rc2? ATM I'm trying this version, but without success. here is my config: destination d_mongo { mongodb( database("test") collection("messages") datetime("some.field.name") value-pairs( scope("selected-macros" "nv-pairs") exclude("R_*") exclude("S_*") exclude("HOST_FROM") exclude("LEGACY_MSGHDR") exclude("MSG") exclude("SDATA") pair("DATE" "$UNIXTIME") pair("some.field.name", "$UNIXTIME") ) ); }; I got the following error: Error parsing afmongodb, inner-dest plugin datetime not found in /etc/syslog-ng/syslog-ng.conf at line 247, column 3: datetime("some.field.name") ^^^^^^^^ In the draft docu for 3.4 I also can't find some hints for this. I run ./configure --enable-ipv6 --enable-tcp-wrapper --disable-spoof-source --without-compile-date --enable-ssl --enable-pcre --enable-capabilities --enable-mongodb Do I something wrong? TIA, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.