[syslog-ng] MongoDB destination driver

[Balint Kovacs]

On 01/02/2011 12:51 AM, Gergely Nagy wrote:
>> This is why I asked in my earlier mail if it's possible to set up the
>> mongo driver to log all vars in a message or a subset of vars in a
>> message. I was hoping it'd be possible for the schema to change somewhat
>> dynamically based on what's present in the messages.
> You can set it up to log a set of vars, and it will only actually
> insert the non-empty values.
>
> Say, if you have something like this:
>
> destination d_mongo {
>    mongodb(
>      keys("host", "program", "pid", "message")
>      values("$HOST", "$PROGRAM", "$PID", "$MSGONLY")
>    );
> };
>
> If a message does not contain a PID, then that will not be added to
> the document, only the rest.
>
> Thus, if you set a maximum of vars, that'll do just what you need, and
> only add those that do have a value.
>
> To the best of my knowledge it is not possible to log all available
> variables (that would be bad too, since there are overlapping macros),
> but you can set up a selected maximum set, and the driver will Do The
> Right Thing, and only store those parts of it, that are set.
Hi,

first of all, thanks for the great work.

I agree with Matthew, that it would be really important to make this 
driver "dynamic", as it would be a great tool combined with patterndb 
for reporting without the need to pre-define fields and a dozen of 
destination statements.

It is actually not that hard to achieve (again, syslog-ng is a breeze), 
pdbtool does quite the same when emitting all variables, the 
nv_table_foreach() function is there to iterate over all of the 
name-value pairs.

However the NVTable struct stores the builtin and dynamic values 
separately and with a small copy-paste coding in nvtable.c you can grab 
only the dynamic values.

Please find a patch attached that introduces the flags() option for the 
mongodb driver and the auto_nvpairs flag, that inserts all dynamic 
name-value pairs into the DB as well. I'm sure that there's a better way 
to implement some parts of it, so please somebody review and clean up if 
possible :)

Usage would look something like this:

destination d_mongo {
   mongodb(
     database("logs")
     keys("host", "program", "pid", "message")
     values("$HOST", "$PROGRAM", "$PID", "$MSGONLY")
     flags(auto_nvpairs)
   );
};

No performance measurements were done yet, I would be glad to see it on 
the same box and same settings as the previous ones. Of course this will 
be a bit slower, ad it makes sense only if you use it in conjunction 
with patterndb, but I expect no drastic drop in performance.

(Disclaimer: I am not a developer, this code is far from being ready for 
production, may leak, etc, etc)

Balint
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-afmongodb-added-support-for-inserting-all-dynamic-va.patch
Type: text/x-patch
Size: 0 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110103/625f37a1/attachment.bin