[syslog-ng] character handling in kv-parser

Tue Dec 13 06:25:14 UTC 2016

On Tue, Dec 13, 2016 at 6:23 AM, Scheidler, Balázs <
balazs.scheidler at balabit.com> wrote:

>
>
> On Dec 12, 2016 20:04, "Fekete, Róbert" <robert.fekete at balabit.com> wrote:
>
> Hi Bazsi,
>
> another parser-related idea that came up during a conversation with
> faxmodem, is that
>  - a way to remove the parsed part of the message (this would be possible
> if the parsers would return the entire parsed string as a separate macro),
> and to
>
>
> I dont really understand this. Is it related to kv-parser or parsers in
> general?
>

I think it could work in general for most parsers. The actual usecase was
for the date-parser, where you can parse an arbitrary date, for example,
from the beginning of a non-RFC message, but then it's difficult to parse
the rest of the message, because you don't know where to start. But I've
seen other cases as well where you had to combine different parsers to
parse messages, and this could be useful.

>
>
>  - delete a leading part of the message (skip until a specific character,
> for example), though now that I think of it, this could be probably done
> with a well-aimed scl that uses the csv-parser.
>
>
> Hmmm is this something like the marker() option for json-parser()?
>

Rather a positional marker I think. Typically, you would combine it with
the previous feature: parse the beginning, check the length of the parsed
part, delete/split at its position, then use a different parser on the
rest. It could also be an offset to the template option of the parser
(like, instead of template(MESSAGE), use template(MESSAGE
offset($lentgh($already-parsed-part))))

>
>
>
> Regards,
>
> Robert
>
> On Mon, Dec 12, 2016 at 5:09 PM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
>> Hi,
>>
>> In Pr #1225 I have added one more option to kv-parser(),
>> extract-stray-words() that would put things the parser didnt recognize in a
>> separate name value pair.
>>
>>
>>
>> On Dec 11, 2016 22:31, "David Boles (dboles)" <dboles at micron.com> wrote:
>>
>>> Great - thank you for confirming what I thought I was seeing. Having '.'
>>> in the allowed range would be great but I can work around the issue.
>>> Another question - from the Linux kernel my log messages have the following
>>> form (from /dev/kmesg):
>>>
>>>
>>> 3,2402,677449975886,-;[MSE] test log message in /home/dboles/...:
>>> Error at foo.c:23: errno 12
>>>  mse_space=kernel
>>>  mse_version=1.0.0-r0.20161210.f9b5a91.dirty
>>>  mse_branch=frog
>>>  mse_0_category=mse_error
>>>  mse_0_version=0
>>>  mse_0_code=12
>>>  mse_0_file=foo.c
>>>  mse_0_line=23
>>>  mse_0_description=errno 12
>>>
>>> The kv-parser works great to obtain the key/value pairs from the above
>>> and $(format-json ...) lets me output them just fine. The next thing I face
>>> is that I want the "human message" part as well but when I refer to
>>> $MESSAGE and friends they always have the whole message. I believe the
>>> $(python ...) mechanism will allow me to snip out what I want, which is:
>>>
>>> [MSE] test log message in /home/dboles/...: Error at foo.c:23: errno 12
>>>
>>> I was curious if there was a better pattern or approach.
>>>
>>> Thanks,
>>>
>>>  - db
>>>
>>>
>>>
>>> ________________________________
>>> From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of
>>> Balazs Scheidler <bazsi77 at gmail.com>
>>> Sent: Sunday, December 11, 2016 12:13 AM
>>> To: Syslog-ng users' and developers' mailing list
>>> Subject: Re: [syslog-ng] character handling in kv-parser
>>>
>>> Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty
>>> important to identify keys in odd formats.
>>>
>>> We can change the set of characters pretty easily but for now it is
>>> hardcoded.
>>>
>>> I have a pending patchset that adds '.' as a char, as I needed that, and
>>> that's what syslog-ng uses to separate key portions.
>>>
>>> Would using a dot work for you? Then it would ve pretty easy to do the
>>> json conversion right within syslog-ng.
>>>
>>>
>>> Something like:
>>>
>>> parser { kv-parser(); };
>>> destination { file("foobar" template("$(format-json *)\n")); };
>>>
>>> Format json would generate embedded objects at dot boundaries.
>>>
>>> On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles at micron.com<mailto:
>>> dboles at micron.com>> wrote:
>>> I am trying to process structured data coming from the Linux kernel's
>>> printk_emit() function which shows up on /dev/kmesg. Since printk_emit()
>>> forces you to flatten hierarchical data into key/value pairs I use a
>>> character delimiter between pieces of the key. So, with the '!' character
>>> as a separator something that looks like { "cat": { "family" : "mammal" } }
>>> would become "cat!family=mammal".
>>>
>>> When I use a kv-parser, and I use characters like '.', '$', '!', '+' or
>>> '^' as a separator the parser seems to just discard everything in the key
>>> except the last part. Alphabetic characters and '_' appear to work - but
>>> they are the most likely to conflict with other strings.
>>>
>>> Is there some way to tell the kv-parser to only consider '=' as special
>>> (or whatever the separator was defined to be) and keep my keys intact?
>>>
>>> Thanks,
>>>
>>>  - db
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=
> syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20161213/b41f5e1f/attachment.html>