[syslog-ng] MongoDB destination driver

[Gergely Nagy]

> It is not just patterndb that can generate dynamic values in a log
> message, so I'd prefer it to be put in the same level as all the other
> values.
> 
> I understand that the user would need some means to select which nv
> pairs need to be added to the document and also with this operation she
> also needs a means to select a whole set, not just single values, which
> syslog-ng doesn't have right now. This would be useful for both mongodb,
> SQL and probably some other formats too.
> 
> Any ideas?

This was covered in one of your earlier mails aswell, but since then, I
had a few more ideas, so I'll reiterate (mind you, my knowledge of
nvtables and what it is used for is lacking):

One can already easily 'filter' the standard stuff with templates (as
there's not many of them, and the set is known and finite - thus listing
all the ones one wants is an acceptable option).

What is missing, is a way to globally restrict what nvtable pairs the
various driver instances see.

The most flexible solution - in my opinion - would be to have an
iterator function in nvtables that can filter keys, based on various
user-settable criteria.

Then, one could have a config like the one you mentioned earlier:

filter-keys('.snmp.*' ltrim ('.snmp.') prefix('foo.'));

This would get parsed into an appropriate structure, and would get
passed down to the filtering function.

The various drivers - mongodb, SQL and whatever else where it's
appropriate - could then use this filtering mechanism.

The filtering itself could be implemented with either fnmatch(), which
is reasonably fast I believe, or whatever other way. It'd even be
possible to add flags later on, so one can choose between shell glob
based filtering, regexes, or whatever else we come up with in the
future.

The hardest part in my opinion, is to figure out how to parse and store
what actions one needs to take on the keys.

As a first step, I'd suggest doing only filtering, and adding the rest
of the features gradually.

I'm fairly certain I can come up with a prototype by monday (provided
nothing unexpected happens during the mongodb driver porting - but that
is almost complete by now).

-- 
|8]