[syslog-ng] character handling in kv-parser

[David Boles (dboles)]

Great - thank you for confirming what I thought I was seeing. Having '.' in the allowed range would be great but I can work around the issue. Another question - from the Linux kernel my log messages have the following form (from /dev/kmesg):


3,2402,677449975886,-;[MSE] test log message in /home/dboles/...: Error at foo.c:23: errno 12
 mse_space=kernel
 mse_version=1.0.0-r0.20161210.f9b5a91.dirty
 mse_branch=frog
 mse_0_category=mse_error
 mse_0_version=0
 mse_0_code=12
 mse_0_file=foo.c
 mse_0_line=23
 mse_0_description=errno 12

The kv-parser works great to obtain the key/value pairs from the above and $(format-json ...) lets me output them just fine. The next thing I face is that I want the "human message" part as well but when I refer to $MESSAGE and friends they always have the whole message. I believe the $(python ...) mechanism will allow me to snip out what I want, which is:

[MSE] test log message in /home/dboles/...: Error at foo.c:23: errno 12

I was curious if there was a better pattern or approach.

Thanks,

 - db



________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77 at gmail.com>
Sent: Sunday, December 11, 2016 12:13 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] character handling in kv-parser

Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.

We can change the set of characters pretty easily but for now it is hardcoded.

I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.

Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.


Something like:

parser { kv-parser(); };
destination { file("foobar" template("$(format-json *)\n")); };

Format json would generate embedded objects at dot boundaries.

On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles at micron.com<mailto:dboles at micron.com>> wrote:
I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".

When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.

Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?

Thanks,

 - db

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq