[syslog-ng] MongoDB destination driver

[Martin Holste]

Cool.  The code must indeed be pretty low-overhead.  I suspect the
main difference in insert rates is probably that one amount fits in
RAM while larger message counts do not.

On Tue, Jan 4, 2011 at 5:05 PM, Gergely Nagy <algernon at balabit.hu> wrote:
> On Tue, 2011-01-04 at 20:20 +0100, Gergely Nagy wrote:
>> On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote:
>> > > #1: Insert the root document, update with dynamic values
>>
>> > This will hit my branch sometime tonight, at which point I'll redo the
>> > benchmark tests.
>>
>> Done! And implemented in such a way that the static keys which one can
>> specify in the keys() option can also contain dots, and they'll be
>> handled properly (ie, turned into neat sub-documents).
>>
>
>> destination d_mongodb {
>>       mongodb(
>>               dynamic_values("dyn")
>>               keys("date", "host", "log.facility", "log.level",
>>                      "program.name", "program.pid", "message")
>>               values("$DATE", "$HOST", "$FACILITY", "$LEVEL",
>>                        "$PROGRAM", "$PID", "$MSGONLY")
>>       );
>> };
>
>
> Using this block, a completely non-scientific test:
>
> * Inserting sshd login messages:
>  + non-capped, non-indexed collection: 12k msg/sec
>  + capped (10Gb, 1k msgs), non-indexed collection: 3k msg/sec
>  + capped (1Mb, 100msg), non-indexed collection: 10k msg/sec
> * Inserting loggen generated messages:
>  + non-capped, non-indexed collection: 12.2k msg/sec
>  + capped (10Gb, 1k msgs), non-indexed collection: 3.1k msg/sec
>  + capped (1Mb, 100msg), non-indexed collection: 10.1k msg/sec
>  + capped (10k, 100msgs), non-indexed collection: 15k msg/sec
>    (this one is terribly useless, but included for the sake of it)
>
> They're within a margin of error, which means that dynamic values do not
> add a significant overhead by the looks of it. The numbers are pretty
> much the same as when I tested without dyn. values a few days ago.
>
> Mind you, these benchmarks are completely non-scientific, done on my
> desktop, running a ton of other things at the same time.
>
> Do note how the cap sizes affect performance: you gotta choose the
> appropriate one, if you're going for capping, otherwise performance
> plummets.
>
> My 100 loggen messages used up roughly 42k space, so setting the cap at
> 50k/100msgs would yield the best results, I suppose. But, that's mongodb
> tuning, which - thankfully - is none of my business :D
>
> --
> |8]
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>