On Tue, 2011-01-04 at 13:39 +0100, Gergely Nagy wrote:
#1: Insert the root document, update with dynamic values
This will hit my branch sometime tonight, at which point I'll redo the benchmark tests.
Done! And implemented in such a way that the static keys which one can specify in the keys() option can also contain dots, and they'll be handled properly (ie, turned into neat sub-documents). Thus, with a block like this: destination d_mongodb { mongodb( dynamic_values("dyn") keys("date", "host", "log.facility", "log.level", "program.name", "program.pid", "message") values("$DATE", "$HOST", "$FACILITY", "$LEVEL", "$PROGRAM", "$PID", "$MSGONLY") ); }; We can end up with a log entry like this: { "_id" : ObjectId("4d2370879d864e560000000a"), "date" : "Jan 4 20:09:59", "dyn" : { "classifier" : { "class" : "system", "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c" }, "secevt" : { "verdict" : "ACCEPT" }, "usracct" : { "application" : "sshd", "authmethod" : "publickey for algernon from 127.0.0.1 port 33659 ssh2", "device" : "127.0.0.1 port 33659 ssh2", "service" : "ssh2", "sessionid" : "10424", "type" : "login", "username" : "algernon from 127.0.0.1 port 33659 ssh2" } }, "host" : "localhost", "log" : { "facility" : "auth", "level" : "info" }, "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2", "program" : { "name" : "sshd", "pid" : "10424" } } Beautiful, isn't it? (And yes, my patterndb rules are still horrid; I'll fix them before I run the benchmarks) And to show you the queries:
db.logs.find().count() 4 db.logs.find({"dyn.usracct.application": "sshd"}, {date: 1, host: 1, log: 1, "dyn.classifier.class": 1, message: 1, "dyn.secevt": 1}) { "_id" : ObjectId("4d2370879d864e560000000a"), "date" : "Jan 4 20:09:59", "dyn" : { "classifier" : { "class" : "system" }, "secevt" : { "verdict" : "ACCEPT" } }, "host" : "localhost", "log" : { "facility" : "auth", "level" : "info" }, "message" : "Accepted publickey for algernon from 127.0.0.1 port 33659 ssh2" } { "_id" : ObjectId("4d2371689d864e560000000d"), "date" : "Jan 4 20:13:44", "dyn" : { "classifier" : { "class" : "system" } }, "host" : "localhost", "log" : { "facility" : "authpriv", "level" : "info" }, "message" : "pam_unix(sshd:session): session closed for user algernon" }
Simply awesome. Thanks to everyone who contributed ideas and nudged me into the right direction! -- |8]