"patterndb" : { ".classifier.class" : "system", ".classifier.rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c", "usracct.authmethod" : "publickey for algernon from ::1 port 59690 ssh2", "usracct.username" : "algernon from ::1 port 59690 ssh2", "usracct.device" : "::1 port 59690 ssh2", "usracct.service" : "ssh2", "usracct.type" : "login", "usracct.sessionid" : "12674", "usracct.application" : "sshd", "secevt.verdict" : "ACCEPT" }
should really look like this:
"patterndb" : { "classifier": { "class" : "system", "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c" }, "usracct": { "authmethod" : "publickey for algernon from ::1 port 59690 ssh2", "username" : "algernon from ::1 port 59690 ssh2", "device" : "::1 port 59690 ssh2", "service" : "ssh2", "type" : "login", "sessionid" : "12674", "application" : "sshd", }, "secevt":{ "verdict" : "ACCEPT" } }
I recognize, however, that this is not a trivial conversion. As a start, just doing a simple substitution of "." for "_" on keys would probably work just fine.
For the time being, the current tip of my branch converts . and $ to _ in dynamic key names (which is stricter than what mongodb allows, but it was simpler to implement it this way). I also have an idea about how to convert the stuff to a well structured format. Actually, I have a few ideas, all with pros and cons: #1: Insert the root document, update with dynamic values We would insert the root document first, up to and including the patterndb: {} sub document. Then we'd iterate over the keys, and use mongodb's update method to add the rest of the stuff:
db.logs.update({_id: <id>}, {$set: {"patterndb.classifier.class": "system"}})
This has the upside of being almost trivial to implement, but has three notable flaws: it will result in more network traffic, and inserting a log message will not be atomic, since the dynamic values are added one at a time. It also has a good chance of fragmenting the database (though, mongodb is said to be clever enough to leave some padding space for objects to grow, which might save us in this case). It is also possible to do bulk updates, like this:
db.logs.update({_id: <id>}, {$set: {"patterndb.classifier.class": "system", "patterndb.classifier.rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"}, "patterndb.secevt.verdict": "ACCEPT"} })
With this, we can reduce the whole operation to two steps: inserting the first, static content, then the dynamic values. However, all of the mentioned flaws remain even with this, they're just not as serious as if we'd insert one by one. #2: Construct the whole document within syslog-ng This has the upside of keeping network traffic to a minimum, and inserts will remain atomic. The downside is that I have no idea how to implement this properly and reliably yet. And my gut feeling is, that whatever solution I end up with, this method will be considerably slower and would require more processing power. #3: Keep the status quo and leave it unstructured. No extra work required on either side, and the values are still reasonably easily queryable. I'll implement #1 tonight, and make it so that one can choose between that and #3, for example with a flag(dynamic_values_restructure) or somesuch option. Gotta find a decent name for the flag, though. -- |8]