Wow, this is amazing! Now the following is possible: web interface drives patterndb XML file creation, created file is automatically loaded by syslog-ng and new patterns are implemented in parsing, new key/value pairs are automatically logged correctly into mongo. So, a fully dynamic parsing solution now exists with a database backend which requires no destination configuration changes. Even the column indexes are dynamic so that new keys are automatically indexed. Not too shabby! On Tue, Jan 4, 2011 at 1:20 PM, Gergely Nagy <algernon@balabit.hu> wrote:
On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote:
#1: Insert the root document, update with dynamic values
This will hit my branch sometime tonight, at which point I'll redo the benchmark tests.
Done! And implemented in such a way that the static keys which one can specify in the keys() option can also contain dots, and they'll be handled properly (ie, turned into neat sub-documents).
Thus, with a block like this:
destination d_mongodb { mongodb( dynamic_values("dyn") keys("date", "host", "log.facility", "log.level", "program.name", "program.pid", "message") values("$DATE", "$HOST", "$FACILITY", "$LEVEL", "$PROGRAM", "$PID", "$MSGONLY") ); };
We can end up with a log entry like this:
{ "_id" : ObjectId("4d2370879d864e560000000a"), "date" : "Jan 4 20:09:59", "dyn" : { "classifier" : { "class" : "system", "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c" }, "secevt" : { "verdict" : "ACCEPT" }, "usracct" : { "application" : "sshd", "authmethod" : "publickey for algernon from 127.0.0.1 port 33659 ssh2", "device" : "127.0.0.1 port 33659 ssh2", "service" : "ssh2", "sessionid" : "10424", "type" : "login", "username" : "algernon from 127.0.0.1 port 33659 ssh2" } }, "host" : "localhost", "log" : { "facility" : "auth", "level" : "info" }, "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2", "program" : { "name" : "sshd", "pid" : "10424" } }
Beautiful, isn't it? (And yes, my patterndb rules are still horrid; I'll fix them before I run the benchmarks)
And to show you the queries:
db.logs.find().count() 4 db.logs.find({"dyn.usracct.application": "sshd"}, {date: 1, host: 1, log: 1, "dyn.classifier.class": 1, message: 1, "dyn.secevt": 1}) { "_id" : ObjectId("4d2370879d864e560000000a"), "date" : "Jan 4 20:09:59", "dyn" : { "classifier" : { "class" : "system" }, "secevt" : { "verdict" : "ACCEPT" } }, "host" : "localhost", "log" : { "facility" : "auth", "level" : "info" }, "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2" } { "_id" : ObjectId("4d2371689d864e560000000d"), "date" : "Jan 4 20:13:44", "dyn" : { "classifier" : { "class" : "system" } }, "host" : "localhost", "log" : { "facility" : "authpriv", "level" : "info" }, "message" : "pam_unix(sshd:session): session closed for user algernon" }
Simply awesome. Thanks to everyone who contributed ideas and nudged me into the right direction!
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html