On Tue, 2011-01-04 at 20:20 +0100, Gergely Nagy wrote:
On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote:
#1: Insert the root document, update with dynamic values
This will hit my branch sometime tonight, at which point I'll redo the benchmark tests.
Done! And implemented in such a way that the static keys which one can specify in the keys() option can also contain dots, and they'll be handled properly (ie, turned into neat sub-documents).
destination d_mongodb { mongodb( dynamic_values("dyn") keys("date", "host", "log.facility", "log.level", "program.name", "program.pid", "message") values("$DATE", "$HOST", "$FACILITY", "$LEVEL", "$PROGRAM", "$PID", "$MSGONLY") ); };
Using this block, a completely non-scientific test: * Inserting sshd login messages: + non-capped, non-indexed collection: 12k msg/sec + capped (10Gb, 1k msgs), non-indexed collection: 3k msg/sec + capped (1Mb, 100msg), non-indexed collection: 10k msg/sec * Inserting loggen generated messages: + non-capped, non-indexed collection: 12.2k msg/sec + capped (10Gb, 1k msgs), non-indexed collection: 3.1k msg/sec + capped (1Mb, 100msg), non-indexed collection: 10.1k msg/sec + capped (10k, 100msgs), non-indexed collection: 15k msg/sec (this one is terribly useless, but included for the sake of it) They're within a margin of error, which means that dynamic values do not add a significant overhead by the looks of it. The numbers are pretty much the same as when I tested without dyn. values a few days ago. Mind you, these benchmarks are completely non-scientific, done on my desktop, running a ton of other things at the same time. Do note how the cap sizes affect performance: you gotta choose the appropriate one, if you're going for capping, otherwise performance plummets. My 100 loggen messages used up roughly 42k space, so setting the cap at 50k/100msgs would yield the best results, I suppose. But, that's mongodb tuning, which - thankfully - is none of my business :D -- |8]