btw: this is the branch that adds '.' as a possible key delimiter: https://github.com/balabit/syslog-ng/pull/1271 On Sun, Dec 11, 2016 at 7:13 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:

Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.

We can change the set of characters pretty easily but for now it is hardcoded.

I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.

Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.

Something like:

parser { kv-parser(); }; destination { file("foobar" template("$(format-json *)\n")); };

Format json would generate embedded objects at dot boundaries.

On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles@micron.com> wrote:

...

I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".

When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.

Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?

Thanks,

- db

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-- Bazsi

Hi, two ideas: * I think you can replace the ! with . using the replace-delimiter() template function ( https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... ) * If your messages do not contain curly braces outside the JSON part, then you can split the message at the first { character using a csv-parser or a pattern database rule to get the text-only part. Regards, Robert On Sun, Dec 11, 2016 at 10:31 PM, David Boles (dboles) <dboles@micron.com> wrote:

Great - thank you for confirming what I thought I was seeing. Having '.' in the allowed range would be great but I can work around the issue. Another question - from the Linux kernel my log messages have the following form (from /dev/kmesg):

3,2402,677449975886,-;[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12 mse_space=kernel mse_version=1.0.0-r0.20161210.f9b5a91.dirty mse_branch=frog mse_0_category=mse_error mse_0_version=0 mse_0_code=12 mse_0_file=foo.c mse_0_line=23 mse_0_description=errno 12

The kv-parser works great to obtain the key/value pairs from the above and $(format-json ...) lets me output them just fine. The next thing I face is that I want the "human message" part as well but when I refer to $MESSAGE and friends they always have the whole message. I believe the $(python ...) mechanism will allow me to snip out what I want, which is:

[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12

I was curious if there was a better pattern or approach.

Thanks,

- db

________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77@gmail.com> Sent: Sunday, December 11, 2016 12:13 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] character handling in kv-parser

Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.

We can change the set of characters pretty easily but for now it is hardcoded.

I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.

Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.

Something like:

parser { kv-parser(); }; destination { file("foobar" template("$(format-json *)\n")); };

Format json would generate embedded objects at dot boundaries.

On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles@micron.com<mailto: dboles@micron.com>> wrote: I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".

When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.

Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?

Thanks,

- db

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Hi Bazsi, another parser-related idea that came up during a conversation with faxmodem, is that - a way to remove the parsed part of the message (this would be possible if the parsers would return the entire parsed string as a separate macro), and to - delete a leading part of the message (skip until a specific character, for example), though now that I think of it, this could be probably done with a well-aimed scl that uses the csv-parser. Regards, Robert On Mon, Dec 12, 2016 at 5:09 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:

Hi,

In Pr #1225 I have added one more option to kv-parser(), extract-stray-words() that would put things the parser didnt recognize in a separate name value pair.

On Dec 11, 2016 22:31, "David Boles (dboles)" <dboles@micron.com> wrote:

...

Great - thank you for confirming what I thought I was seeing. Having '.' in the allowed range would be great but I can work around the issue. Another question - from the Linux kernel my log messages have the following form (from /dev/kmesg):

3,2402,677449975886,-;[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12 mse_space=kernel mse_version=1.0.0-r0.20161210.f9b5a91.dirty mse_branch=frog mse_0_category=mse_error mse_0_version=0 mse_0_code=12 mse_0_file=foo.c mse_0_line=23 mse_0_description=errno 12

The kv-parser works great to obtain the key/value pairs from the above and $(format-json ...) lets me output them just fine. The next thing I face is that I want the "human message" part as well but when I refer to $MESSAGE and friends they always have the whole message. I believe the $(python ...) mechanism will allow me to snip out what I want, which is:

[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12

I was curious if there was a better pattern or approach.

Thanks,

- db

________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77@gmail.com> Sent: Sunday, December 11, 2016 12:13 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] character handling in kv-parser

Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.

We can change the set of characters pretty easily but for now it is hardcoded.

I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.

Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.

Something like:

parser { kv-parser(); }; destination { file("foobar" template("$(format-json *)\n")); };

Format json would generate embedded objects at dot boundaries.

On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles@micron.com<mailto: dboles@micron.com>> wrote: I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".

When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.

Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?

Thanks,

- db

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

On Tue, Dec 13, 2016 at 6:23 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:

On Dec 12, 2016 20:04, "Fekete, Róbert" <robert.fekete@balabit.com> wrote:

Hi Bazsi,

another parser-related idea that came up during a conversation with faxmodem, is that - a way to remove the parsed part of the message (this would be possible if the parsers would return the entire parsed string as a separate macro), and to

I dont really understand this. Is it related to kv-parser or parsers in general?

I think it could work in general for most parsers. The actual usecase was for the date-parser, where you can parse an arbitrary date, for example, from the beginning of a non-RFC message, but then it's difficult to parse the rest of the message, because you don't know where to start. But I've seen other cases as well where you had to combine different parsers to parse messages, and this could be useful.

- delete a leading part of the message (skip until a specific character, for example), though now that I think of it, this could be probably done with a well-aimed scl that uses the csv-parser.

Hmmm is this something like the marker() option for json-parser()?

Rather a positional marker I think. Typically, you would combine it with the previous feature: parse the beginning, check the length of the parsed part, delete/split at its position, then use a different parser on the rest. It could also be an offset to the template option of the parser (like, instead of template(MESSAGE), use template(MESSAGE offset($lentgh($already-parsed-part))))

Regards,

Robert

On Mon, Dec 12, 2016 at 5:09 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:

...

Hi,

In Pr #1225 I have added one more option to kv-parser(), extract-stray-words() that would put things the parser didnt recognize in a separate name value pair.

On Dec 11, 2016 22:31, "David Boles (dboles)" <dboles@micron.com> wrote:

...

Great - thank you for confirming what I thought I was seeing. Having '.' in the allowed range would be great but I can work around the issue. Another question - from the Linux kernel my log messages have the following form (from /dev/kmesg):

3,2402,677449975886,-;[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12 mse_space=kernel mse_version=1.0.0-r0.20161210.f9b5a91.dirty mse_branch=frog mse_0_category=mse_error mse_0_version=0 mse_0_code=12 mse_0_file=foo.c mse_0_line=23 mse_0_description=errno 12

The kv-parser works great to obtain the key/value pairs from the above and $(format-json ...) lets me output them just fine. The next thing I face is that I want the "human message" part as well but when I refer to $MESSAGE and friends they always have the whole message. I believe the $(python ...) mechanism will allow me to snip out what I want, which is:

[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12

I was curious if there was a better pattern or approach.

Thanks,

- db

________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77@gmail.com> Sent: Sunday, December 11, 2016 12:13 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] character handling in kv-parser

Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.

We can change the set of characters pretty easily but for now it is hardcoded.

I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.

Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.

Something like:

parser { kv-parser(); }; destination { file("foobar" template("$(format-json *)\n")); };

Format json would generate embedded objects at dot boundaries.

On Dec 11, 2016 00:03, "David Boles (dboles)" <dboles@micron.com<mailto: dboles@micron.com>> wrote: I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".

When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.

Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?

Thanks,

- db

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq