[syslog-ng] MongoDB destination driver

[Martin Holste]

Wow, this is amazing!  Now the following is possible: web interface
drives patterndb XML file creation, created file is automatically
loaded by syslog-ng and new patterns are implemented in parsing, new
key/value pairs are automatically logged correctly into mongo.  So, a
fully dynamic parsing solution now exists with a database backend
which requires no destination configuration changes.  Even the column
indexes are dynamic so that new keys are automatically indexed.  Not
too shabby!

On Tue, Jan 4, 2011 at 1:20 PM, Gergely Nagy <algernon at balabit.hu> wrote:
> On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote:
>> > #1: Insert the root document, update with dynamic values
>
>> This will hit my branch sometime tonight, at which point I'll redo the
>> benchmark tests.
>
> Done! And implemented in such a way that the static keys which one can
> specify in the keys() option can also contain dots, and they'll be
> handled properly (ie, turned into neat sub-documents).
>
> Thus, with a block like this:
>
> destination d_mongodb {
>        mongodb(
>                dynamic_values("dyn")
>                keys("date", "host", "log.facility", "log.level",
>                     "program.name", "program.pid", "message")
>                values("$DATE", "$HOST", "$FACILITY", "$LEVEL",
>                       "$PROGRAM", "$PID", "$MSGONLY")
>        );
> };
>
> We can end up with a log entry like this:
>
> { "_id" : ObjectId("4d2370879d864e560000000a"),
>  "date" : "Jan  4 20:09:59",
>  "dyn" : {
>    "classifier" : {
>      "class" : "system",
>      "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"
>    },
>    "secevt" : { "verdict" : "ACCEPT" },
>    "usracct" : {
>      "application" : "sshd",
>      "authmethod" : "publickey for algernon from 127.0.0.1 port 33659 ssh2",
>      "device" : "127.0.0.1 port 33659 ssh2",
>      "service" : "ssh2",
>      "sessionid" : "10424",
>      "type" : "login",
>      "username" : "algernon from 127.0.0.1 port 33659 ssh2"
>    }
>  },
>  "host" : "localhost",
>  "log" : { "facility" : "auth", "level" : "info" },
>  "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2",
>  "program" : { "name" : "sshd", "pid" : "10424" }
> }
>
> Beautiful, isn't it? (And yes, my patterndb rules are still horrid; I'll
> fix them before I run the benchmarks)
>
> And to show you the queries:
>
>> db.logs.find().count()
> 4
>> db.logs.find({"dyn.usracct.application": "sshd"},
>               {date: 1, host: 1, log: 1, "dyn.classifier.class": 1, message: 1, "dyn.secevt": 1})
> { "_id" : ObjectId("4d2370879d864e560000000a"), "date" : "Jan  4 20:09:59",
>  "dyn" : { "classifier" : { "class" : "system" }, "secevt" : { "verdict" : "ACCEPT" } },
>  "host" : "localhost",
>  "log" : { "facility" : "auth", "level" : "info" },
>  "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2" }
> { "_id" : ObjectId("4d2371689d864e560000000d"), "date" : "Jan  4 20:13:44",
>  "dyn" : { "classifier" : { "class" : "system" } },
>  "host" : "localhost",
>  "log" : { "facility" : "authpriv", "level" : "info" },
>  "message" : "pam_unix(sshd:session): session closed for user algernon" }
>
> Simply awesome. Thanks to everyone who contributed ideas and nudged me
> into the right direction!
>
> --
> |8]
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>