On Dec 12, 2016 20:04, "Fekete, Róbert" <robert.fekete@balabit.com> wrote: Hi Bazsi, another parser-related idea that came up during a conversation with faxmodem, is that - a way to remove the parsed part of the message (this would be possible if the parsers would return the entire parsed string as a separate macro), and to I dont really understand this. Is it related to kv-parser or parsers in general? - delete a leading part of the message (skip until a specific character, for example), though now that I think of it, this could be probably done with a well-aimed scl that uses the csv-parser. Hmmm is this something like the marker() option for json-parser()? Regards, Robert On Mon, Dec 12, 2016 at 5:09 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
In Pr #1225 I have added one more option to kv-parser(), extract-stray-words() that would put things the parser didnt recognize in a separate name value pair.
On Dec 11, 2016 22:31, "David Boles (dboles)" <dboles@micron.com> wrote:
Great - thank you for confirming what I thought I was seeing. Having '.' in the allowed range would be great but I can work around the issue. Another question - from the Linux kernel my log messages have the following form (from /dev/kmesg):
3,2402,677449975886,-;[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12 mse_space=kernel mse_version=1.0.0-r0.20161210.f9b5a91.dirty mse_branch=frog mse_0_category=mse_error mse_0_version=0 mse_0_code=12 mse_0_file=foo.c mse_0_line=23 mse_0_description=errno 12
The kv-parser works great to obtain the key/value pairs from the above and $(format-json ...) lets me output them just fine. The next thing I face is that I want the "human message" part as well but when I refer to $MESSAGE and friends they always have the whole message. I believe the $(python ...) mechanism will allow me to snip out what I want, which is:
[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12
I was curious if there was a better pattern or approach.
Thanks,
- db
________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77@gmail.com> Sent: Sunday, December 11, 2016 12:13 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] character handling in kv-parser
Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.
We can change the set of characters pretty easily but for now it is hardcoded.
I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.
Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.
Something like:
parser { kv-parser(); }; destination { file("foobar" template("$(format-json *)\n")); };
Format json would generate embedded objects at dot boundaries.
On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles@micron.com<mailto: dboles@micron.com>> wrote: I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".
When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.
Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?
Thanks,
- db
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq