It is not just patterndb that can generate dynamic values in a log message, so I'd prefer it to be put in the same level as all the other values.
I understand that the user would need some means to select which nv pairs need to be added to the document and also with this operation she also needs a means to select a whole set, not just single values, which syslog-ng doesn't have right now. This would be useful for both mongodb, SQL and probably some other formats too.
Any ideas?
This was covered in one of your earlier mails aswell, but since then, I had a few more ideas, so I'll reiterate (mind you, my knowledge of nvtables and what it is used for is lacking): One can already easily 'filter' the standard stuff with templates (as there's not many of them, and the set is known and finite - thus listing all the ones one wants is an acceptable option). What is missing, is a way to globally restrict what nvtable pairs the various driver instances see. The most flexible solution - in my opinion - would be to have an iterator function in nvtables that can filter keys, based on various user-settable criteria. Then, one could have a config like the one you mentioned earlier: filter-keys('.snmp.*' ltrim ('.snmp.') prefix('foo.')); This would get parsed into an appropriate structure, and would get passed down to the filtering function. The various drivers - mongodb, SQL and whatever else where it's appropriate - could then use this filtering mechanism. The filtering itself could be implemented with either fnmatch(), which is reasonably fast I believe, or whatever other way. It'd even be possible to add flags later on, so one can choose between shell glob based filtering, regexes, or whatever else we come up with in the future. The hardest part in my opinion, is to figure out how to parse and store what actions one needs to take on the keys. As a first step, I'd suggest doing only filtering, and adding the rest of the features gradually. I'm fairly certain I can come up with a prototype by monday (provided nothing unexpected happens during the mongodb driver porting - but that is almost complete by now). -- |8]