[syslog-ng] MongoDB destination driver

[Gergely Nagy]

> Very cool.  As a stop-gap, one can always pipe to a program() to do
> the actual inserts.  That gives you a chance to batch the logs as a
> TSV and then run mongoimport on the TSV for high-performance inserts.
> You should be able to get around 20k-50k inserts/sec that way.

I plan to do bulk inserts within the driver - much like how the sql
driver does bulk commits with explicit-commit turned on.

The plan is to make a writer thread, which will combine a
(configurable) set of documents and insert them in bulk. I'm halfway
through implementing that, should be done during the weekend.

> The key thing to know when profiling MongoDB inserts is that you need
> to let everything run long enough for Mongo to fill RAM to capacity so
> that it is forced to begin using disk.  Up until that point,
> everything is done in RAM, which means you're not seeing the long-term
> rates, only the burst rates.

Oh, that's nice to know, thanks!