Hi Bazsi,

another parser-related idea that came up during a conversation with faxmodem, is that

 - a way to remove the parsed part of the message (this would be possible if the parsers would return the entire parsed string as a separate macro), and to

 - delete a leading part of the message (skip until a specific character, for example), though now that I think of it, this could be probably done with a well-aimed scl that uses the csv-parser.

Regards,

Robert

On Mon, Dec 12, 2016 at 5:09 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:

Hi,

In Pr #1225 I have added one more option to kv-parser(), extract-stray-words() that would put things the parser didnt recognize in a separate name value pair.

On Dec 11, 2016 22:31, "David Boles (dboles)" <dboles@micron.com> wrote:

Great - thank you for confirming what I thought I was seeing. Having '.' in the allowed range would be great but I can work around the issue. Another question - from the Linux kernel my log messages have the following form (from /dev/kmesg):


3,2402,677449975886,-;[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12
 mse_space=kernel
 mse_version=1.0.0-r0.20161210.f9b5a91.dirty
 mse_branch=frog
 mse_0_category=mse_error
 mse_0_version=0
 mse_0_code=12
 mse_0_file=foo.c
 mse_0_line=23
 mse_0_description=errno 12

The kv-parser works great to obtain the key/value pairs from the above and $(format-json ...) lets me output them just fine. The next thing I face is that I want the "human message" part as well but when I refer to $MESSAGE and friends they always have the whole message. I believe the $(python ...) mechanism will allow me to snip out what I want, which is:

[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12

I was curious if there was a better pattern or approach.

Thanks,

 - db



________________________________
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77@gmail.com>
Sent: Sunday, December 11, 2016 12:13 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] character handling in kv-parser

Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.

We can change the set of characters pretty easily but for now it is hardcoded.

I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.

Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.


Something like:

parser { kv-parser(); };
destination { file("foobar" template("$(format-json *)\n")); };

Format json would generate embedded objects at dot boundaries.

On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles@micron.com<mailto:dboles@micron.com>> wrote:
I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".

When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.

Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?

Thanks,

 - db

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq