[syslog-ng] character handling in kv-parser

[Fekete, Róbert]

Hi,

two ideas:
 * I think  you can replace the ! with . using the replace-delimiter()
template function (
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-template-functions.html
)
 * If your messages do not contain curly braces outside the JSON part, then
you can split the message at the first { character using a csv-parser or a
pattern database rule to get the text-only part.

Regards,

Robert

On Sun, Dec 11, 2016 at 10:31 PM, David Boles (dboles) <dboles at micron.com>
wrote:

> Great - thank you for confirming what I thought I was seeing. Having '.'
> in the allowed range would be great but I can work around the issue.
> Another question - from the Linux kernel my log messages have the following
> form (from /dev/kmesg):
>
>
> 3,2402,677449975886,-;[MSE] test log message in /home/dboles/...:
> Error at foo.c:23: errno 12
>  mse_space=kernel
>  mse_version=1.0.0-r0.20161210.f9b5a91.dirty
>  mse_branch=frog
>  mse_0_category=mse_error
>  mse_0_version=0
>  mse_0_code=12
>  mse_0_file=foo.c
>  mse_0_line=23
>  mse_0_description=errno 12
>
> The kv-parser works great to obtain the key/value pairs from the above and
> $(format-json ...) lets me output them just fine. The next thing I face is
> that I want the "human message" part as well but when I refer to $MESSAGE
> and friends they always have the whole message. I believe the $(python ...)
> mechanism will allow me to snip out what I want, which is:
>
> [MSE] test log message in /home/dboles/...: Error at foo.c:23: errno 12
>
> I was curious if there was a better pattern or approach.
>
> Thanks,
>
>  - db
>
>
>
> ________________________________
> From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Balazs
> Scheidler <bazsi77 at gmail.com>
> Sent: Sunday, December 11, 2016 12:13 AM
> To: Syslog-ng users' and developers' mailing list
> Subject: Re: [syslog-ng] character handling in kv-parser
>
> Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty
> important to identify keys in odd formats.
>
> We can change the set of characters pretty easily but for now it is
> hardcoded.
>
> I have a pending patchset that adds '.' as a char, as I needed that, and
> that's what syslog-ng uses to separate key portions.
>
> Would using a dot work for you? Then it would ve pretty easy to do the
> json conversion right within syslog-ng.
>
>
> Something like:
>
> parser { kv-parser(); };
> destination { file("foobar" template("$(format-json *)\n")); };
>
> Format json would generate embedded objects at dot boundaries.
>
> On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles at micron.com<mailto:
> dboles at micron.com>> wrote:
> I am trying to process structured data coming from the Linux kernel's
> printk_emit() function which shows up on /dev/kmesg. Since printk_emit()
> forces you to flatten hierarchical data into key/value pairs I use a
> character delimiter between pieces of the key. So, with the '!' character
> as a separator something that looks like { "cat": { "family" : "mammal" } }
> would become "cat!family=mammal".
>
> When I use a kv-parser, and I use characters like '.', '$', '!', '+' or
> '^' as a separator the parser seems to just discard everything in the key
> except the last part. Alphabetic characters and '_' appear to work - but
> they are the most likely to conflict with other strings.
>
> Is there some way to tell the kv-parser to only consider '=' as special
> (or whatever the separator was defined to be) and keep my keys intact?
>
> Thanks,
>
>  - db
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20161212/67dd36f0/attachment.html>