Hi again! Last time (http://article.gmane.org/gmane.comp.syslog-ng/10542) I described the way I imagine value_pairs(), both the syntax, and the implementation. Today, I post here because there's a few bits and pieces of code to show: both the generic value_pairs() framework implemented, and a modified afmongodb() driver that uses it, and a modified tfjson template function aswell! There's a few rough edges to sort out still, but it's already usable (see the end of this post for links). Lets have a look at the afmongodb driver! Example: afmongodb ================== destination d_mongo { mongodb(); }; The default settings and all, will include every key-value pair syslog-ng knows about: { "HOST" : "localhost", "HOST_FROM" : "localhost", "LEGACY_MSGHDR" : "sshd[19868]: ", "MESSAGE" : "Accepted publickey for algernon from ::1 port 42248 ssh2", "PID" : "19868", "PROGRAM" : "sshd", "SOURCE" : "s_network", "_id" : ObjectId("4d4418efaef570564d000013"), "classifier" : { "class" : "system", "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c" }, "secevt" : { "verdict" : "ACCEPT" }, "usracct" : { "application" : "sshd", "authmethod" : "publickey", "device" : "::1", "service" : "ssh2", "sessionid" : "19868", "type" : "login", "username" : "algernon" } } All keys equal, the dynamic keys appear on the same level as every other one (unlike the previous solution in mongodb, which put dynamic keys into their own namespace). However, there's a few things here we're not all that interested in, and we'd like to limit the keys to the dynamic keys only, plus some builtins we explicitly specify. We can do that! destination d_mongo { mongodb( value_pairs(builtins(no) select("*") "$HOST" "$MESSAGE" ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME")) ); }; This will include all dynamic values, along with the $HOST and $MESSAGE builtins, and two extra keys: PROGRAM and TIMESTAMP: { "HOST" : "localhost", "MESSAGE" : "Accepted publickey for algernon from ::1 port 47932 ssh2", "PROGRAM" : "sshd[20839]", "TIMESTAMP" : "1296308990", "_id" : ObjectId("4d441afea7423bc050000013"), "classifier" : { "class" : "system", "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c" }, "secevt" : { "verdict" : "ACCEPT" }, "usracct" : { "application" : "sshd", "authmethod" : "publickey", "device" : "::1", "service" : "ssh2", "sessionid" : "20839", "type" : "login", "username" : "algernon" } } That's all nice and good, but we don't really care about classifier.rule_id, do we? destination d_mongo { mongodb( value_pairs(builtins(no) select("*") exclude(".classifier.rule_id") "$HOST" "$MESSAGE" ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME")) ); }; And this will do exactly what it says: skip builtins, select everything that is left, and exclude ".classifier.rule_id" from that, and then add a few extra stuff on our own. Of course, one can have multiple selects and excludes, any number of macros listed, and any number of (key value) pairs. In the future, if and when new types of selects (for example, POSIX or PCRE regexps) are implemented, the select and exclude statements will grow a type() option aswell. TODO ==== There's a few things still in limbo regarding the syntax, however: * Evan Rempel's comment that "$HOST" and the like should always refer to the value of the macro is convincing, so that part of the syntax will change most probably. My current favourite ide is to list the macro name as-is, without quotes, without the dollar sign: value-pairs(HOST MESSAGE ("foo" "bar") ...) But maybe his macro(HOST MESSAGE) suggestion would be better... We'll see, I guess! * The way key-value pairs can be specified looks iffy. Something like pair("key", "$VALUE") would look better. * The parser needs to be usable by - for example - template functions aswell. It's trivial to add value_pairs() support for any destination driver (or source driver - anything that has a grammar file in the sources can easily use this functionality), but at the moment template functions cannot, and tfjson has its own, GOption-based parser, which isn't 100% compatible with the bison/flex parser. For example, the bison/flex parser keeps the order of select() and exclude() statements, so one can do exclude("*") select("usracct.*) and that will select everything under "usracct", but nothing else. With tfjson's GOption-based parser, all selects are evaluated first, and then the excludes, so the above example would lead to no keys selected at all. The Links ========= My git tree: git://git.madhouse-project.org/syslog-ng/syslog-ng-3.3.git (or on the web: http://git.madhouse-project.org/syslog-ng/syslog-ng-3.3/) The interesting branches are: * work/value-pairs/base: The basic value-pairs implementation, without parser support. * work/value-pairs/grammar: bison/flex parser built upon the ValuePairs base. * work/tfjson/value-pairs: tfjson ported to ValuePairs. The syntax is the same as before (see my previous mail about tfjson). * work/afmongodb-vp: afmongodb ported to ValuePairs. Example syntax can be seen above. -- |8]