[syslog-ng] [RFC]: value_pairs() demo

[Gergely Nagy]

Hi again!

Last time (http://article.gmane.org/gmane.comp.syslog-ng/10542) I
described the way I imagine value_pairs(), both the syntax, and the
implementation.

Today, I post here because there's a few bits and pieces of code to
show: both the generic value_pairs() framework implemented, and a
modified afmongodb() driver that uses it, and a modified tfjson template
function aswell!

There's a few rough edges to sort out still, but it's already usable
(see the end of this post for links).

Lets have a look at the afmongodb driver!

Example: afmongodb
==================

destination d_mongo { mongodb(); };

The default settings and all, will include every key-value pair
syslog-ng knows about:

        { "HOST" : "localhost", 
          "HOST_FROM" : "localhost", 
          "LEGACY_MSGHDR" : "sshd[19868]: ", 
          "MESSAGE" : "Accepted publickey for algernon from ::1 port 42248 ssh2",
          "PID" : "19868",
          "PROGRAM" : "sshd", 
          "SOURCE" : "s_network",
          "_id" : ObjectId("4d4418efaef570564d000013"),
          "classifier" : { 
            "class" : "system", 
            "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c"
          },
          "secevt" : { 
            "verdict" : "ACCEPT" 
          }, 
          "usracct" : { 
            "application" : "sshd", 
            "authmethod" : "publickey", 
            "device" : "::1", 
            "service" : "ssh2", 
            "sessionid" : "19868", 
            "type" : "login", 
            "username" : "algernon"
          }
        }

All keys equal, the dynamic keys appear on the same level as every other
one (unlike the previous solution in mongodb, which put dynamic keys
into their own namespace).

However, there's a few things here we're not all that interested in, and
we'd like to limit the keys to the dynamic keys only, plus some builtins
we explicitly specify. We can do that!

destination d_mongo {
  mongodb( 
    value_pairs(builtins(no) select("*") "$HOST" "$MESSAGE"
                ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
  );
};

This will include all dynamic values, along with the $HOST and $MESSAGE
builtins, and two extra keys: PROGRAM and TIMESTAMP:

        { "HOST" : "localhost", 
          "MESSAGE" : "Accepted publickey for algernon from ::1 port 47932 ssh2", 
          "PROGRAM" : "sshd[20839]", 
          "TIMESTAMP" : "1296308990", 
          "_id" : ObjectId("4d441afea7423bc050000013"), 
          "classifier" : { 
            "class" : "system", 
            "rule_id" : "4dd5a329-da83-4876-a431-ddcb59c2858c" 
          }, 
          "secevt" : { "verdict" : "ACCEPT" }, 
          "usracct" : { 
            "application" : "sshd", 
            "authmethod" : "publickey", 
            "device" : "::1", 
            "service" : "ssh2", 
            "sessionid" : "20839", 
            "type" : "login", 
            "username" : "algernon"
          }
        }

That's all nice and good, but we don't really care about
classifier.rule_id, do we?

destination d_mongo {
  mongodb( 
    value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
                "$HOST" "$MESSAGE"
                ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
  );
};

And this will do exactly what it says: skip builtins, select everything
that is left, and exclude ".classifier.rule_id" from that, and then add
a few extra stuff on our own.

Of course, one can have multiple selects and excludes, any number of
macros listed, and any number of (key value) pairs.

In the future, if and when new types of selects (for example, POSIX or
PCRE regexps) are implemented, the select and exclude statements will
grow a type() option aswell.

TODO
====

There's a few things still in limbo regarding the syntax, however:

* Evan Rempel's comment that "$HOST" and the like should always refer to
the value of the macro is convincing, so that part of the syntax will
change most probably.

My current favourite ide is to list the macro name as-is, without
quotes, without the dollar sign:
  value-pairs(HOST MESSAGE ("foo" "bar") ...)

But maybe his macro(HOST MESSAGE) suggestion would be better... We'll
see, I guess!

* The way key-value pairs can be specified looks iffy. Something like
pair("key", "$VALUE") would look better.

* The parser needs to be usable by - for example - template functions
aswell. It's trivial to add value_pairs() support for any destination
driver (or source driver - anything that has a grammar file in the
sources can easily use this functionality), but at the moment template
functions cannot, and tfjson has its own, GOption-based parser, which
isn't 100% compatible with the bison/flex parser.

For example, the bison/flex parser keeps the order of select() and
exclude() statements, so one can do exclude("*") select("usracct.*) and
that will select everything under "usracct", but nothing else. With
tfjson's GOption-based parser, all selects are evaluated first, and then
the excludes, so the above example would lead to no keys selected at
all.

The Links
=========

My git tree: git://git.madhouse-project.org/syslog-ng/syslog-ng-3.3.git
(or on the web:
http://git.madhouse-project.org/syslog-ng/syslog-ng-3.3/)

The interesting branches are:

* work/value-pairs/base: The basic value-pairs implementation, without
parser support.
* work/value-pairs/grammar: bison/flex parser built upon the ValuePairs
base.
* work/tfjson/value-pairs: tfjson ported to ValuePairs. The syntax is
the same as before (see my previous mail about tfjson).
* work/afmongodb-vp: afmongodb ported to ValuePairs. Example syntax can
be seen above.

-- 
|8]