[syslog-ng] MongoDB destination driver

[Balazs Scheidler]

On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote:
> > #1: Insert the root document, update with dynamic values
> > 
> > We would insert the root document first, up to and including the
> > patterndb: {} sub document. Then we'd iterate over the keys, and use
> > mongodb's update method to add the rest of the stuff:
> > 
> > > db.logs.update({_id: <id>}, 
> >      {$set: {"patterndb.classifier.class": "system"}})
> > 
> > This has the upside of being almost trivial to implement, but has three
> > notable flaws: it will result in more network traffic, and inserting a
> > log message will not be atomic, since the dynamic values are added one
> > at a time. It also has a good chance of fragmenting the database
> > (though, mongodb is said to be clever enough to leave some padding space
> > for objects to grow, which might save us in this case).
> > 
> > It is also possible to do bulk updates, like this:
> > 
> > > db.logs.update({_id: <id>},
> >      {$set: {"patterndb.classifier.class": "system",
> >              "patterndb.classifier.rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"},
> >              "patterndb.secevt.verdict": "ACCEPT"}
> >      })
> > 
> > With this, we can reduce the whole operation to two steps: inserting the
> > first, static content, then the dynamic values. However, all of the
> > mentioned flaws remain even with this, they're just not as serious as if
> > we'd insert one by one.
> 
> Good news: we can use upserts and get rid of all the flaws:
> 
> > db.logs.update({_id: <id>},
>                  {$set: {message: "some message", <rest of the static keys>,
>                          "patterndb.classifier.class": "system",
>                          "patterndb.classifier.rule_id": "0xdeadbeef",
>                          "patterndb.secevt.verdict": "ACCEPT"}
>                  }, true)
> 
> We just have to pre-generate the ID, which is luckily easy, as the
> mongodb driver has a function to do just that. In return, we get an
> atomic insert, only one message towards the mongodb server, and no
> fragmentation.
> 
> And it's dead easy to add this to my mongodb destination, since the
> dynamic values are already dot-separated, just the way we want them (I
> only have to strip the leading dots).
> 
> This will hit my branch sometime tonight, at which point I'll redo the
> benchmark tests.
> 

It is not just patterndb that can generate dynamic values in a log
message, so I'd prefer it to be put in the same level as all the other
values.

I understand that the user would need some means to select which nv
pairs need to be added to the document and also with this operation she
also needs a means to select a whole set, not just single values, which
syslog-ng doesn't have right now. This would be useful for both mongodb,
SQL and probably some other formats too.

Any ideas?

-- 
Bazsi