[syslog-ng] MongoDB destination driver

[Balazs Scheidler]

On Fri, 2011-01-14 at 14:40 +0100, Gergely Nagy wrote:
> > It is not just patterndb that can generate dynamic values in a log
> > message, so I'd prefer it to be put in the same level as all the other
> > values.
> > 
> > I understand that the user would need some means to select which nv
> > pairs need to be added to the document and also with this operation she
> > also needs a means to select a whole set, not just single values, which
> > syslog-ng doesn't have right now. This would be useful for both mongodb,
> > SQL and probably some other formats too.
> > 
> > Any ideas?
> 
> This was covered in one of your earlier mails aswell, but since then, I
> had a few more ideas, so I'll reiterate (mind you, my knowledge of
> nvtables and what it is used for is lacking):
> 
> One can already easily 'filter' the standard stuff with templates (as
> there's not many of them, and the set is known and finite - thus listing
> all the ones one wants is an acceptable option).
> 
> What is missing, is a way to globally restrict what nvtable pairs the
> various driver instances see.
> 
> The most flexible solution - in my opinion - would be to have an
> iterator function in nvtables that can filter keys, based on various
> user-settable criteria.
> 
> Then, one could have a config like the one you mentioned earlier:
> 
> filter-keys('.snmp.*' ltrim ('.snmp.') prefix('foo.'));
> 
> This would get parsed into an appropriate structure, and would get
> passed down to the filtering function.
> 
> The various drivers - mongodb, SQL and whatever else where it's
> appropriate - could then use this filtering mechanism.
> 
> The filtering itself could be implemented with either fnmatch(), which
> is reasonably fast I believe, or whatever other way. It'd even be
> possible to add flags later on, so one can choose between shell glob
> based filtering, regexes, or whatever else we come up with in the
> future.
> 
> The hardest part in my opinion, is to figure out how to parse and store
> what actions one needs to take on the keys.
> 
> As a first step, I'd suggest doing only filtering, and adding the rest
> of the features gradually.
> 
> I'm fairly certain I can come up with a prototype by monday (provided
> nothing unexpected happens during the mongodb driver porting - but that
> is almost complete by now).

I might give up on the ability to exchange prefixes of name-value pairs
when expanding them into mongodb documents.

Without that requirement, this feature could be fairly simple, and would
be quite easy to adapt to SQL and welf and probably many other things we
come up in the future.

What this boils down to, is that in addition to providing the current
keys/values options in mongodb (and the similar columns/values in SQL),
we could have a combination of the two:

destination d_mongo {
	mongodb(...
		value-pairs(('host', '$HOST'), 'PROGRAM', '*')
	);
};

There are 3 forms of pairs supported:
  * (name, value): traditional syslog-ng templating, name specifies the
key, value is a syslog-ng template (containing macros)
  * name: name is both the name of the key and the name of the nv-pair
in syslog-ng, in essence, equivalent to ('name', '${name}') described in
the first syntax
  * glob: in this case the result is all the name-value pairs matched by
the glob string, the name of the key is the same as the nvpair in
syslog-ng, e.g. it produces a series of ('name', '${name}') pairs
matching the specified glob.

I don't expect regexps would be needed in this case. NVPairs are
supposed to have structured names after all.

This would be somewhat similar to my $(format-welf) template function
idea described earlier on this mailing list. Although that too provided
exchange-prefix functionality (and assuming we come to a feasible model
that gets adapted to mongodb and/or sql, I'd probably change the idea of
format-welf too, that's especially easy as it isn't implemented
yet :) ).

-- 
Bazsi