[syslog-ng] MongoDB destination driver

[Gergely Nagy]

On Tue, 2011-01-04 at 20:20 +0100, Gergely Nagy wrote: 
> On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote: 
> > > #1: Insert the root document, update with dynamic values
> 
> > This will hit my branch sometime tonight, at which point I'll redo the
> > benchmark tests.
> 
> Done! And implemented in such a way that the static keys which one can
> specify in the keys() option can also contain dots, and they'll be
> handled properly (ie, turned into neat sub-documents).
> 

> destination d_mongodb {
> 	mongodb(
> 		dynamic_values("dyn")
>     		keys("date", "host", "log.facility", "log.level", 
>                      "program.name", "program.pid", "message")
> 		values("$DATE", "$HOST", "$FACILITY", "$LEVEL",
>                        "$PROGRAM", "$PID", "$MSGONLY")
> 	);
> };


Using this block, a completely non-scientific test:

* Inserting sshd login messages:
  + non-capped, non-indexed collection: 12k msg/sec
  + capped (10Gb, 1k msgs), non-indexed collection: 3k msg/sec
  + capped (1Mb, 100msg), non-indexed collection: 10k msg/sec
* Inserting loggen generated messages:
  + non-capped, non-indexed collection: 12.2k msg/sec
  + capped (10Gb, 1k msgs), non-indexed collection: 3.1k msg/sec
  + capped (1Mb, 100msg), non-indexed collection: 10.1k msg/sec
  + capped (10k, 100msgs), non-indexed collection: 15k msg/sec
    (this one is terribly useless, but included for the sake of it)

They're within a margin of error, which means that dynamic values do not
add a significant overhead by the looks of it. The numbers are pretty
much the same as when I tested without dyn. values a few days ago.

Mind you, these benchmarks are completely non-scientific, done on my
desktop, running a ton of other things at the same time.

Do note how the cap sizes affect performance: you gotta choose the
appropriate one, if you're going for capping, otherwise performance
plummets.

My 100 loggen messages used up roughly 42k space, so setting the cap at
50k/100msgs would yield the best results, I suppose. But, that's mongodb
tuning, which - thankfully - is none of my business :D

-- 
|8]