[syslog-ng] MongoDB destination driver

[Matthew Hall]

On Thu, Dec 30, 2010 at 08:11:07PM +0100, Gergely Nagy wrote:
>   $ git clone -b algernon/dest/mongodb
> git://git.madhouse-project.org/syslog-ng/syslog-ng-3.2.git

> It is using the MongoDB C client library
> (http://www.mongodb.org/display/DOCS/C+Language+Center) - I simply
> embedded the sources for now, lacking a better option. Once compiled,
> one can already begin using it with the default options:
> 
>   destination d_mongodb { mongodb(); };
> 
> This will try to connect to localhost:27017, and use the logs
> collection in the syslog-ng database, and will log all the standard
> fields. Of course, all of those are configurable!
> 
> To demonstrate all the - currently - available options, the
> destination definition above is the same as the following:
> 
>   destination d_mongodb {
>     mongodb(
>       host("localhost")
>       port(27017)
>       database("syslog-ng")
>       collection("logs")
>       keys("date", "facility", "level", "host", "program", "pid", "message")
>       values("${R_YEAR}-${R_MONTH}-${R_DAY}
> ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST",
> "$PROGRAM", "$PID", "$MSGONLY")
>     );
>   };
> 
> A few things, like authentication and some template options are not
> configurable yet, partly because I didn't figure out what they're good
> for, or how they work. But I will get there at some point, especially
> if there's interest in said features.

> Hope you like the code, and perhaps find it useful!

Good work. I am wondering if support for MongoDB must be added to the 
core code or if it could also be added as a libdbi driver which could be 
used in more than just syslog-ng.

I am wondering if it would be possible to take advantage of MongoDB's 
dynamic nature, and log all of the defined name-value pairs in a 
message, or of a list of name-value pairs.

This would eliminate a common problem of trying to get the most possible 
message fields into the DB without wasting space on empty ones, which 
plagues many of us when using relational DBs to store large quantities 
of often times dissimilar messages from different devices or software 
applications.

Matthew.