[syslog-ng] MongoDB destination driver

[Gergely Nagy]

It's been a while I spam^Wnotified the list with mongodb updates, so the
recent news from the past few days are as follows:

* The syslog-ng 3.2 version is abandoned, obsolete.
* The driver has been ported to 3.3, and the database writer was split
out into a separate thread.
* The underlying mongodb connector was replaced: threw out the former,
and implemented my own (for various reasons, which are too numerous to
list here). The new mongodb connector will be maintained as a separate
project, but a version of it will be embedded into syslog-ng for
convenience's sake. The canonical repo for it is over at github:
https://github.com/algernon/libmongo-client

While this might not seem much, the above changes brought a few good
side effects:

* If we loose connection to MongoDB, for one reason or the other, the
driver does not drop messages immediately anymore: we have a queue
(configurable via log_fifo_size()).
* Database writes are in a separate thread, thus, network latency does
not affect the speed at which the driver can accept messages.
* Due to the new underlying connector, we handle error cases a lot
better. As in, we don't throw up and abort(). Though, error handling
still has to improve, but the driver's in a much better state.
* There's support for $SEQNUM in templates now. Though I can't really
imagine a situation where I could use it, but I needed them for ObjectID
generation anyway, so why not expose them to the templates aswell?

However, there's one downside too: we lost authentication support
(temporarily, until I re-implement that in the connector).

The speed is about the same as 3.2's, threading didn't improve our
performance (well, it did, if we consider network latency, but my tests
are local) - I didn't expect that to improve to begin with.

I'm actually surprised that despite changing the connector library, and
porting over to threads, the driver still maintained its speed.

My project page should be (mostly) up to date, but for the adventurous:

  git clone -b modules/afmongodb git://git.madhouse-project.org/syslog-ng/syslog-ng-3.3.git

The modules/afmongodb branch will hopefully be quiet during the weekend.

-- 
|8]