On Sat, 2011-01-01 at 14:24 -0600, Martin Holste wrote:
Super cool! At those rates, I think few will benefit from the bulk insert benefits, so I'd put that low on the feature priority list, especially with the opportunity to create bugs with the complexity. My main feature to add (aside from the two you mentioned already on the roadmap) would be a way to use the keys from a patterndb database so that the db and collection in Mongo stay the same, but the key names change with every patterndb rule. That's really the big payoff with Mongo--you don't have to define a rigid schema, so you don't have to know the column names ahead of time. That's a big deal considering that the patterndb can change on the fly. Being confined to predefined templates in the config limits the potential. Bazsi, any idea how to do this?
sorry for not answering any sooner, I was skimming through these emails, but never had the time to actually think about this stuff. we would definitely need a way to query the contents of a message in a structured way. e.g. if a message is a set of name-value pairs, it'd be nice to select a subset of those NV pairs in a single operation, in order to put them to a structured output format. for instance with either mongodb or sql, it'd make sense to put all name-value pairs starting with a given prefix to the output in a single operation. for example: mongodb(nv-pairs(".snmp.*")) Which would select a set of nv pairs from the message and put them in keys. A kind of name-transformation would be useful too: mongodb(nv-pairs(".snmp.*" ltrim('.snmp.') prefix('foo.')) Which would result in all NV pairs with a name beginning with .snmp. to become foo prefixed. the same could be applied when formatting WELF logs, perhaps would also be useful in rewrite rules. hmm.. maybe I should refresh my XSLT memories to see how this looks like in XPath/XQuery. -- Bazsi