[syslog-ng] character handling in kv-parser

[Fekete, Róbert]

Hi Bazsi,

another parser-related idea that came up during a conversation with
faxmodem, is that
 - a way to remove the parsed part of the message (this would be possible
if the parsers would return the entire parsed string as a separate macro),
and to
 - delete a leading part of the message (skip until a specific character,
for example), though now that I think of it, this could be probably done
with a well-aimed scl that uses the csv-parser.

Regards,

Robert

On Mon, Dec 12, 2016 at 5:09 PM, Scheidler, Balázs <
balazs.scheidler at balabit.com> wrote:

> Hi,
>
> In Pr #1225 I have added one more option to kv-parser(),
> extract-stray-words() that would put things the parser didnt recognize in a
> separate name value pair.
>
>
>
> On Dec 11, 2016 22:31, "David Boles (dboles)" <dboles at micron.com> wrote:
>
>> Great - thank you for confirming what I thought I was seeing. Having '.'
>> in the allowed range would be great but I can work around the issue.
>> Another question - from the Linux kernel my log messages have the following
>> form (from /dev/kmesg):
>>
>>
>> 3,2402,677449975886,-;[MSE] test log message in /home/dboles/...:
>> Error at foo.c:23: errno 12
>>  mse_space=kernel
>>  mse_version=1.0.0-r0.20161210.f9b5a91.dirty
>>  mse_branch=frog
>>  mse_0_category=mse_error
>>  mse_0_version=0
>>  mse_0_code=12
>>  mse_0_file=foo.c
>>  mse_0_line=23
>>  mse_0_description=errno 12
>>
>> The kv-parser works great to obtain the key/value pairs from the above
>> and $(format-json ...) lets me output them just fine. The next thing I face
>> is that I want the "human message" part as well but when I refer to
>> $MESSAGE and friends they always have the whole message. I believe the
>> $(python ...) mechanism will allow me to snip out what I want, which is:
>>
>> [MSE] test log message in /home/dboles/...: Error at foo.c:23: errno 12
>>
>> I was curious if there was a better pattern or approach.
>>
>> Thanks,
>>
>>  - db
>>
>>
>>
>> ________________________________
>> From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Balazs
>> Scheidler <bazsi77 at gmail.com>
>> Sent: Sunday, December 11, 2016 12:13 AM
>> To: Syslog-ng users' and developers' mailing list
>> Subject: Re: [syslog-ng] character handling in kv-parser
>>
>> Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty
>> important to identify keys in odd formats.
>>
>> We can change the set of characters pretty easily but for now it is
>> hardcoded.
>>
>> I have a pending patchset that adds '.' as a char, as I needed that, and
>> that's what syslog-ng uses to separate key portions.
>>
>> Would using a dot work for you? Then it would ve pretty easy to do the
>> json conversion right within syslog-ng.
>>
>>
>> Something like:
>>
>> parser { kv-parser(); };
>> destination { file("foobar" template("$(format-json *)\n")); };
>>
>> Format json would generate embedded objects at dot boundaries.
>>
>> On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles at micron.com<mailto:
>> dboles at micron.com>> wrote:
>> I am trying to process structured data coming from the Linux kernel's
>> printk_emit() function which shows up on /dev/kmesg. Since printk_emit()
>> forces you to flatten hierarchical data into key/value pairs I use a
>> character delimiter between pieces of the key. So, with the '!' character
>> as a separator something that looks like { "cat": { "family" : "mammal" } }
>> would become "cat!family=mammal".
>>
>> When I use a kv-parser, and I use characters like '.', '$', '!', '+' or
>> '^' as a separator the parser seems to just discard everything in the key
>> except the last part. Alphabetic characters and '_' appear to work - but
>> they are the most likely to conflict with other strings.
>>
>> Is there some way to tell the kv-parser to only consider '=' as special
>> (or whatever the separator was defined to be) and keep my keys intact?
>>
>> Thanks,
>>
>>  - db
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20161212/8fa7328c/attachment.html>