[syslog-ng] MongoDB destination driver

Gergely Nagy algernon at balabit.hu
Tue Jan 4 20:20:28 CET 2011


On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote: 
> > #1: Insert the root document, update with dynamic values

> This will hit my branch sometime tonight, at which point I'll redo the
> benchmark tests.

Done! And implemented in such a way that the static keys which one can
specify in the keys() option can also contain dots, and they'll be
handled properly (ie, turned into neat sub-documents).

Thus, with a block like this:

destination d_mongodb {
	mongodb(
		dynamic_values("dyn")
    		keys("date", "host", "log.facility", "log.level", 
                     "program.name", "program.pid", "message")
		values("$DATE", "$HOST", "$FACILITY", "$LEVEL",
                       "$PROGRAM", "$PID", "$MSGONLY")
	);
};

We can end up with a log entry like this:

{ "_id" : ObjectId("4d2370879d864e560000000a"), 
  "date" : "Jan  4 20:09:59",
  "dyn" : { 
    "classifier" : { 
      "class" : "system", 
      "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"
    }, 
    "secevt" : { "verdict" : "ACCEPT" },
    "usracct" : {
      "application" : "sshd", 
      "authmethod" : "publickey for algernon from 127.0.0.1 port 33659 ssh2", 
      "device" : "127.0.0.1 port 33659 ssh2", 
      "service" : "ssh2", 
      "sessionid" : "10424", 
      "type" : "login", 
      "username" : "algernon from 127.0.0.1 port 33659 ssh2" 
    } 
  }, 
  "host" : "localhost", 
  "log" : { "facility" : "auth", "level" : "info" }, 
  "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2", 
  "program" : { "name" : "sshd", "pid" : "10424" }
}

Beautiful, isn't it? (And yes, my patterndb rules are still horrid; I'll
fix them before I run the benchmarks)

And to show you the queries:

> db.logs.find().count()
4
> db.logs.find({"dyn.usracct.application": "sshd"}, 
               {date: 1, host: 1, log: 1, "dyn.classifier.class": 1, message: 1, "dyn.secevt": 1})
{ "_id" : ObjectId("4d2370879d864e560000000a"), "date" : "Jan  4 20:09:59",
  "dyn" : { "classifier" : { "class" : "system" }, "secevt" : { "verdict" : "ACCEPT" } }, 
  "host" : "localhost",
  "log" : { "facility" : "auth", "level" : "info" }, 
  "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2" }
{ "_id" : ObjectId("4d2371689d864e560000000d"), "date" : "Jan  4 20:13:44", 
  "dyn" : { "classifier" : { "class" : "system" } }, 
  "host" : "localhost", 
  "log" : { "facility" : "authpriv", "level" : "info" }, 
  "message" : "pam_unix(sshd:session): session closed for user algernon" }

Simply awesome. Thanks to everyone who contributed ideas and nudged me
into the right direction!

-- 
|8]





More information about the syslog-ng mailing list