#1: Insert the root document, update with dynamic values
We would insert the root document first, up to and including the patterndb: {} sub document. Then we'd iterate over the keys, and use mongodb's update method to add the rest of the stuff:
db.logs.update({_id: <id>}, {$set: {"patterndb.classifier.class": "system"}})
This has the upside of being almost trivial to implement, but has three notable flaws: it will result in more network traffic, and inserting a log message will not be atomic, since the dynamic values are added one at a time. It also has a good chance of fragmenting the database (though, mongodb is said to be clever enough to leave some padding space for objects to grow, which might save us in this case).
It is also possible to do bulk updates, like this:
db.logs.update({_id: <id>}, {$set: {"patterndb.classifier.class": "system", "patterndb.classifier.rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"}, "patterndb.secevt.verdict": "ACCEPT"} })
With this, we can reduce the whole operation to two steps: inserting the first, static content, then the dynamic values. However, all of the mentioned flaws remain even with this, they're just not as serious as if we'd insert one by one.
Good news: we can use upserts and get rid of all the flaws:
db.logs.update({_id: <id>}, {$set: {message: "some message", <rest of the static keys>, "patterndb.classifier.class": "system", "patterndb.classifier.rule_id": "0xdeadbeef", "patterndb.secevt.verdict": "ACCEPT"} }, true)
We just have to pre-generate the ID, which is luckily easy, as the mongodb driver has a function to do just that. In return, we get an atomic insert, only one message towards the mongodb server, and no fragmentation. And it's dead easy to add this to my mongodb destination, since the dynamic values are already dot-separated, just the way we want them (I only have to strip the leading dots). This will hit my branch sometime tonight, at which point I'll redo the benchmark tests. -- |8]