[syslog-ng] MongoDB destination driver

Martin Holste mcholste at gmail.com
Tue Jan 4 17:47:48 CET 2011


This is very good, especially since Mongo really simplifies creating
indexes.  Specifically, you can create an index on the "patterndb" key
in your example message, and it will automatically index all subkeys
and values.  See the manual page here:
http://www.mongodb.org/display/DOCS/Using+Multikeys+to+Simulate+a+Large+Number+of+Indexes
for the specific example.  Moreover, you can choose to index only
certain subkeys to save inserting effort and disk space.  All of this
lends itself very nicely to patterndb.

On Tue, Jan 4, 2011 at 6:39 AM, Gergely Nagy <algernon at balabit.hu> wrote:
>> #1: Insert the root document, update with dynamic values
>>
>> We would insert the root document first, up to and including the
>> patterndb: {} sub document. Then we'd iterate over the keys, and use
>> mongodb's update method to add the rest of the stuff:
>>
>> > db.logs.update({_id: <id>},
>>      {$set: {"patterndb.classifier.class": "system"}})
>>
>> This has the upside of being almost trivial to implement, but has three
>> notable flaws: it will result in more network traffic, and inserting a
>> log message will not be atomic, since the dynamic values are added one
>> at a time. It also has a good chance of fragmenting the database
>> (though, mongodb is said to be clever enough to leave some padding space
>> for objects to grow, which might save us in this case).
>>
>> It is also possible to do bulk updates, like this:
>>
>> > db.logs.update({_id: <id>},
>>      {$set: {"patterndb.classifier.class": "system",
>>              "patterndb.classifier.rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"},
>>              "patterndb.secevt.verdict": "ACCEPT"}
>>      })
>>
>> With this, we can reduce the whole operation to two steps: inserting the
>> first, static content, then the dynamic values. However, all of the
>> mentioned flaws remain even with this, they're just not as serious as if
>> we'd insert one by one.
>
> Good news: we can use upserts and get rid of all the flaws:
>
>> db.logs.update({_id: <id>},
>                 {$set: {message: "some message", <rest of the static keys>,
>                         "patterndb.classifier.class": "system",
>                         "patterndb.classifier.rule_id": "0xdeadbeef",
>                         "patterndb.secevt.verdict": "ACCEPT"}
>                 }, true)
>
> We just have to pre-generate the ID, which is luckily easy, as the
> mongodb driver has a function to do just that. In return, we get an
> atomic insert, only one message towards the mongodb server, and no
> fragmentation.
>
> And it's dead easy to add this to my mongodb destination, since the
> dynamic values are already dot-separated, just the way we want them (I
> only have to strip the leading dots).
>
> This will hit my branch sometime tonight, at which point I'll redo the
> benchmark tests.
>
> --
> |8]
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>


More information about the syslog-ng mailing list