[syslog-ng] MongoDB destination driver

[Gergely Nagy]

On Fri, 2011-01-14 at 12:56 +0100, Balazs Scheidler wrote:
> On Sat, 2011-01-01 at 14:24 -0600, Martin Holste wrote:
> > Super cool!  At those rates, I think few will benefit from the bulk
> > insert benefits, so I'd put that low on the feature priority list,
> > especially with the opportunity to create bugs with the complexity.
> > My main feature to add (aside from the two you mentioned already on
> > the roadmap) would be a way to use the keys from a patterndb database
> > so that the db and collection in Mongo stay the same, but the key
> > names change with every patterndb rule.  That's really the big payoff
> > with Mongo--you don't have to define a rigid schema, so you don't have
> > to know the column names ahead of time.  That's a big deal considering
> > that the patterndb can change on the fly.  Being confined to
> > predefined templates in the config limits the potential.  Bazsi, any
> > idea how to do this?
> 
> sorry for not answering any sooner, I was skimming through these emails,
> but never had the time to actually think about this stuff.
> 
> we would definitely need a way to query the contents of a message in a
> structured way.
> 
> e.g. if a message is a set of name-value pairs, it'd be nice to select a
> subset of those NV pairs in a single operation, in order to put them to
> a structured output format.
> 
> for instance with either mongodb or sql, it'd make sense to put all
> name-value pairs starting with a given prefix to the output in a single
> operation.
> 
> for example:
> 
> mongodb(nv-pairs(".snmp.*"))
> 
> Which would select a set of nv pairs from the message and put them in
> keys. A kind of name-transformation would be useful too:
> 
> mongodb(nv-pairs(".snmp.*" ltrim('.snmp.') prefix('foo.'))
> 
> Which would result in all NV pairs with a name beginning with .snmp. to
> become foo prefixed.

Ugh, that'd be a bit tricky to implement, but if shell glob-like syntax
is an acceptable solution, I can make that work with relative ease - at
least the selection part.

Rewriting the keys would be another thing, something much trickier.

However, both of these should - in my opinion - be implemented outside
of the mongodb driver, so other destinations can use them in the future
(if someone comes up with a driver for, say, couchdb, that could use
this too; I can even imagine the sql destination using this...).

A general filtering solution for nvtables would probably do the trick.

The pattern matching can be done by, say, fnmatch(), but a better option
would be to allow filtering nvtable keys by an arbitrary function - I'm
not familiar with nvtables however, will take a closer look as soon as
possible.

That same function could be used to rewrite keys...

Lets say, we have a function like this:

gchar *nvtables_filter_cb (const gchar *key, gpointer user_data);

We could then implement a filter that filters out (returns NULL) for
anything that doesn't match .snmp.*, and if a key does match, it
strips .snmp. and prefixes with 'foo.'.

Wouldn't even be too hard to accomplish, I believe.

But alas, this is in my opinion outside of the scope of the mongodb
driver, and would be an independent feature of syslog-ng. One that the
driver would certainly benefit from.

I'll happily sit down and code a few ideas up, once I'm comfortable with
the mongodb driver itself (adding extra features once the driver is
stable would be my preferred course of action :).

-- 
|8]

PS: Funny how I get from "tricky to implement" to "hey, I'll code it up
in a weekend" within 2 minutes. I love how syslog-ng makes this possible
=)