[syslog-ng] tfjson: JSON-generating template function

[Balazs Scheidler]

Hi,

This is cool stuff, thanks. Hopefully I get to integrate it soon.

On Fri, 2011-01-21 at 17:27 +0100, Gergely Nagy wrote:
> Hi!
> 
> Looks like the list didn't like me sending the whole patch, so I'll have
> to omit the patch part this time, and hope that the wall of text below
> won't hit the size limit.
> 
> Based on the work of Balint Kovacs <blint at balabit.hu>, a $(format_json)
> template function was implemented: with this function, one can hammer
> log messages into JSON format.
> 
> While that is not immediately useful, we have a few things planned that
> will - with a bit of luck - be able to use this function.
> 
> The function itself, without arguments, will reformat the log message
> into JSON. For example, a loggen-generated message will look like this
> (formatting added by me):
> 
>   { "HOST": "localhost", "HOST_FROM": "localhost", 
>     "MESSAGE": "seq: 0000000000, thread: 0000, runid: 1295625842, stamp: 2011-01-21T17:04:02 PADD...",
>     "PROGRAM": "prg00000", "PID": "1234", "SOURCE": "s_tcp",
>     "LEGACY_MSGHDR": "prg00000[1234]: ", ".classifier.class": "unknown" }
> 
> While this is all cool, there's more! The function accepts parameters:
> '--select <GLOB>' and '--exclude <GLOB>', where <GLOB> is a shell-like
> glob pattern. One can have any number of selects or globs. The way this
> works, is that if a key doesn't match any of the select globs, it will
> not be included. If it does, but matches any of the exclude globs, it
> will be dropped. Otherwise it's included. The default select glob is
> "*", which is why everything is included by default.
> 
> The globs are matched against the keys: macro names without the dollar
> sign, patterndb keys, and so on.
> 
> With this knowledge, we can tweak the template like this:
> 
>   $(format_json --select .classifier.* --select usracct.* --exclude *.*id)
> 
> Then, given the following message:
> 
>   <86>Jan 21 17:10:21 moria sshd[31930]: pam_unix(sshd:session): session closed for user algernon
> 
> We'll end up with this JSON:
> 
>   { ".classifier.class": "system", "usracct.username": "algernon", 
>     "usracct.type": "logout", "usracct.application": "sshd" }
> 
> Now, that's not very useful, as we're lacking the HOST and a whole lot
> of other things. Adding all that as selects would be a bit excessive,
> therefore one can list macro names (without the dollar sign) along with
> the select/exclude options, and they'll be included too.
> 
> Furthermore, one can list key=value pairs, for example, using the log
> message quite like the one above, and the following template:
> 
>   $(format_json -s .classifier.* -s usracct.* -x *.*id HOST MESSAGE prg='$PROGRAM[$PID]')
> 
> We'll end up with:
> 
>   { ".classifier.class": "system", "usracct.username": "algernon",
>     "usracct.type": "logout", "usracct.application": "sshd",
>     "MESSAGE": "pam_unix(sshd:session): session closed for user algernon",
>     "prg": "sshd[32216]", "HOST": "localhost" }
> 
> At the moment, only strings are supported, and there is no way to
> structure the JSON properly yet. We'll figure out a way to do that too,
> eventually. There's a few more things in the pipeline for tfjson, so
> treat this version as a beta.
> 
> The code is available on the integration/tfjson/base branch of my
> repository at git://git.madhouse-project.org/syslog-ng/syslog-ng-3.3.git
> 
> Browsable online at:
> http://git.madhouse-project.org/syslog-ng/syslog-ng-3.3/tree/modules/tfjson/tfjson.c?h=integration/tfjson/base
> 
> The bulk of the work was done by Balint, I just happened to accidentally
> refactor it a tiny bit while reviewing his code.
> 

-- 
Bazsi