[syslog-ng] MongoDB destination driver

[Gergely Nagy]

> #1: Insert the root document, update with dynamic values
> 
> We would insert the root document first, up to and including the
> patterndb: {} sub document. Then we'd iterate over the keys, and use
> mongodb's update method to add the rest of the stuff:
> 
> > db.logs.update({_id: <id>}, 
>      {$set: {"patterndb.classifier.class": "system"}})
> 
> This has the upside of being almost trivial to implement, but has three
> notable flaws: it will result in more network traffic, and inserting a
> log message will not be atomic, since the dynamic values are added one
> at a time. It also has a good chance of fragmenting the database
> (though, mongodb is said to be clever enough to leave some padding space
> for objects to grow, which might save us in this case).
> 
> It is also possible to do bulk updates, like this:
> 
> > db.logs.update({_id: <id>},
>      {$set: {"patterndb.classifier.class": "system",
>              "patterndb.classifier.rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"},
>              "patterndb.secevt.verdict": "ACCEPT"}
>      })
> 
> With this, we can reduce the whole operation to two steps: inserting the
> first, static content, then the dynamic values. However, all of the
> mentioned flaws remain even with this, they're just not as serious as if
> we'd insert one by one.

Good news: we can use upserts and get rid of all the flaws:

> db.logs.update({_id: <id>},
                 {$set: {message: "some message", <rest of the static keys>,
                         "patterndb.classifier.class": "system",
                         "patterndb.classifier.rule_id": "0xdeadbeef",
                         "patterndb.secevt.verdict": "ACCEPT"}
                 }, true)

We just have to pre-generate the ID, which is luckily easy, as the
mongodb driver has a function to do just that. In return, we get an
atomic insert, only one message towards the mongodb server, and no
fragmentation.

And it's dead easy to add this to my mongodb destination, since the
dynamic values are already dot-separated, just the way we want them (I
only have to strip the leading dots).

This will hit my branch sometime tonight, at which point I'll redo the
benchmark tests.

-- 
|8]