[syslog-ng] mongodb and TTL collections

[Gergely Nagy]

Gergely Nagy <algernon at balabit.hu> writes:

> "Lucas, Sascha" <Sascha.Lucas at gisa.de> writes:
>
>> Hi,
>>
>> Mongodb 2.2 has a new feature called TTL collections (expire data by
>> setting a TTL). An analogy to the classic logrotate. One constraint
>> is, that you index a field of date BSON type[1]. As far as I know, atm
>> syslog-ng can only store strings? Or how can it be done to store one
>> field as date?
>
> Right now, only with an external program. With syslog-ng 3.3, it will
> stay that way, for 3.4, I have a half-baked patch that adds a date()
> option to the mongodb destination.
>
> I will backport that patch to 3.3 aswell, and it will be available as a
> separate patch, but not as part of the official release.
>
> If all goes well, this will be done by this weekend. Hopefully this'll
> do until syslog-ng learns how to transfer type information too (which is
> something I want to do for 3.4 too, but may not have time for it).

This took a little more than the next weekend (I really should stop
trying to estimate, it hardly ever works), but I just sent a patch to
the list for 3.4 that allows one to do this:

mongodb(datetime("some.field.name")
        value-pairs(pair("some.field.name", "$UNIXTIME")));

That is, whatever you put in as the value of datetime(), will be
searched for in the list of value pairs, and if a field is found, its
value will be converted to proper datetime-typed thing. The code assumes
that the value is a unix timestamp, and will not insert the value if it
fails the conversion.

I have not backported it to 3.3 yet, as the code there is considerably
different, but a similar approach can be taken there too - and it will
be even simpler to achieve the same thing. Nevertheless, I'll do the
backport in a couple of minutes, the results will be available on my
feature/3.3/mongodb/datetime branch shortly after.

-- 
|8]