On Fri, 2011-01-14 at 12:56 +0100, Balazs Scheidler wrote:
On Sat, 2011-01-01 at 14:24 -0600, Martin Holste wrote:
Super cool! At those rates, I think few will benefit from the bulk insert benefits, so I'd put that low on the feature priority list, especially with the opportunity to create bugs with the complexity. My main feature to add (aside from the two you mentioned already on the roadmap) would be a way to use the keys from a patterndb database so that the db and collection in Mongo stay the same, but the key names change with every patterndb rule. That's really the big payoff with Mongo--you don't have to define a rigid schema, so you don't have to know the column names ahead of time. That's a big deal considering that the patterndb can change on the fly. Being confined to predefined templates in the config limits the potential. Bazsi, any idea how to do this?
sorry for not answering any sooner, I was skimming through these emails, but never had the time to actually think about this stuff.
we would definitely need a way to query the contents of a message in a structured way.
e.g. if a message is a set of name-value pairs, it'd be nice to select a subset of those NV pairs in a single operation, in order to put them to a structured output format.
for instance with either mongodb or sql, it'd make sense to put all name-value pairs starting with a given prefix to the output in a single operation.
for example:
mongodb(nv-pairs(".snmp.*"))
Which would select a set of nv pairs from the message and put them in keys. A kind of name-transformation would be useful too:
mongodb(nv-pairs(".snmp.*" ltrim('.snmp.') prefix('foo.'))
Which would result in all NV pairs with a name beginning with .snmp. to become foo prefixed.
Ugh, that'd be a bit tricky to implement, but if shell glob-like syntax is an acceptable solution, I can make that work with relative ease - at least the selection part. Rewriting the keys would be another thing, something much trickier. However, both of these should - in my opinion - be implemented outside of the mongodb driver, so other destinations can use them in the future (if someone comes up with a driver for, say, couchdb, that could use this too; I can even imagine the sql destination using this...). A general filtering solution for nvtables would probably do the trick. The pattern matching can be done by, say, fnmatch(), but a better option would be to allow filtering nvtable keys by an arbitrary function - I'm not familiar with nvtables however, will take a closer look as soon as possible. That same function could be used to rewrite keys... Lets say, we have a function like this: gchar *nvtables_filter_cb (const gchar *key, gpointer user_data); We could then implement a filter that filters out (returns NULL) for anything that doesn't match .snmp.*, and if a key does match, it strips .snmp. and prefixes with 'foo.'. Wouldn't even be too hard to accomplish, I believe. But alas, this is in my opinion outside of the scope of the mongodb driver, and would be an independent feature of syslog-ng. One that the driver would certainly benefit from. I'll happily sit down and code a few ideas up, once I'm comfortable with the mongodb driver itself (adding extra features once the driver is stable would be my preferred course of action :). -- |8] PS: Funny how I get from "tricky to implement" to "hey, I'll code it up in a weekend" within 2 minutes. I love how syslog-ng makes this possible =)