Cool. The code must indeed be pretty low-overhead. I suspect the main difference in insert rates is probably that one amount fits in RAM while larger message counts do not. On Tue, Jan 4, 2011 at 5:05 PM, Gergely Nagy <algernon@balabit.hu> wrote:
On Tue, 2011-01-04 at 20:20 +0100, Gergely Nagy wrote:
On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote:
#1: Insert the root document, update with dynamic values
This will hit my branch sometime tonight, at which point I'll redo the benchmark tests.
Done! And implemented in such a way that the static keys which one can specify in the keys() option can also contain dots, and they'll be handled properly (ie, turned into neat sub-documents).
destination d_mongodb { mongodb( dynamic_values("dyn") keys("date", "host", "log.facility", "log.level", "program.name", "program.pid", "message") values("$DATE", "$HOST", "$FACILITY", "$LEVEL", "$PROGRAM", "$PID", "$MSGONLY") ); };
Using this block, a completely non-scientific test:
* Inserting sshd login messages: + non-capped, non-indexed collection: 12k msg/sec + capped (10Gb, 1k msgs), non-indexed collection: 3k msg/sec + capped (1Mb, 100msg), non-indexed collection: 10k msg/sec * Inserting loggen generated messages: + non-capped, non-indexed collection: 12.2k msg/sec + capped (10Gb, 1k msgs), non-indexed collection: 3.1k msg/sec + capped (1Mb, 100msg), non-indexed collection: 10.1k msg/sec + capped (10k, 100msgs), non-indexed collection: 15k msg/sec (this one is terribly useless, but included for the sake of it)
They're within a margin of error, which means that dynamic values do not add a significant overhead by the looks of it. The numbers are pretty much the same as when I tested without dyn. values a few days ago.
Mind you, these benchmarks are completely non-scientific, done on my desktop, running a ton of other things at the same time.
Do note how the cap sizes affect performance: you gotta choose the appropriate one, if you're going for capping, otherwise performance plummets.
My 100 loggen messages used up roughly 42k space, so setting the cap at 50k/100msgs would yield the best results, I suppose. But, that's mongodb tuning, which - thankfully - is none of my business :D
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html