[syslog-ng] MongoDB destination driver

[Martin Holste]

Very cool.  As a stop-gap, one can always pipe to a program() to do
the actual inserts.  That gives you a chance to batch the logs as a
TSV and then run mongoimport on the TSV for high-performance inserts.
You should be able to get around 20k-50k inserts/sec that way.

The key thing to know when profiling MongoDB inserts is that you need
to let everything run long enough for Mongo to fill RAM to capacity so
that it is forced to begin using disk.  Up until that point,
everything is done in RAM, which means you're not seeing the long-term
rates, only the burst rates.

On Thu, Dec 30, 2010 at 2:51 PM, Gergely Nagy
<algernon at madhouse-project.org> wrote:
>> Good work. I am wondering if support for MongoDB must be added to the
>> core code or if it could also be added as a libdbi driver which could be
>> used in more than just syslog-ng.
>
> While the inserting code is very similar, and could be added to
> libdbi, the query code is very different. I do not think that adding
> it to libdbi would work.
>
> But to be honest, even the inserting code is different enough to make
> it tricky at best, to add it to libdbi.
>
>> I am wondering if it would be possible to take advantage of MongoDB's
>> dynamic nature, and log all of the defined name-value pairs in a
>> message, or of a list of name-value pairs.
>
> Not adding empty values is certainly possible, I'll add support for
> that shortly. Thanks for the suggestion!
>
> It already is possible to log only selected name-value pairs:
>
> destination d_mongodb { mongodb(keys("host", "message")
> values("$HOST", "$MSGONLY")); };
>
> Though, if one is empty, it will still be added to the store at the
> moment, but like I said, I'll fix that shortly.
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>