[syslog-ng] MongoDB destination driver

[Balazs Scheidler]

On Sat, 2011-01-01 at 14:24 -0600, Martin Holste wrote:
> Super cool!  At those rates, I think few will benefit from the bulk
> insert benefits, so I'd put that low on the feature priority list,
> especially with the opportunity to create bugs with the complexity.
> My main feature to add (aside from the two you mentioned already on
> the roadmap) would be a way to use the keys from a patterndb database
> so that the db and collection in Mongo stay the same, but the key
> names change with every patterndb rule.  That's really the big payoff
> with Mongo--you don't have to define a rigid schema, so you don't have
> to know the column names ahead of time.  That's a big deal considering
> that the patterndb can change on the fly.  Being confined to
> predefined templates in the config limits the potential.  Bazsi, any
> idea how to do this?

sorry for not answering any sooner, I was skimming through these emails,
but never had the time to actually think about this stuff.

we would definitely need a way to query the contents of a message in a
structured way.

e.g. if a message is a set of name-value pairs, it'd be nice to select a
subset of those NV pairs in a single operation, in order to put them to
a structured output format.

for instance with either mongodb or sql, it'd make sense to put all
name-value pairs starting with a given prefix to the output in a single
operation.

for example:

mongodb(nv-pairs(".snmp.*"))

Which would select a set of nv pairs from the message and put them in
keys. A kind of name-transformation would be useful too:

mongodb(nv-pairs(".snmp.*" ltrim('.snmp.') prefix('foo.'))

Which would result in all NV pairs with a name beginning with .snmp. to
become foo prefixed.

the same could be applied when formatting WELF logs, perhaps would also
be useful in rewrite rules.

hmm.. maybe I should refresh my XSLT memories to see how this looks like
in XPath/XQuery.

-- 
Bazsi