<div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Dec 12, 2016 20:04, "Fekete, Róbert" <<a href="mailto:robert.fekete@balabit.com">robert.fekete@balabit.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi Bazsi, <br><br></div>another parser-related idea that came up during a conversation with faxmodem, is that <br></div> - a way to remove the parsed part of the message (this would be possible if the parsers would return the entire parsed string as a separate macro), and to<br></div></div></div></div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">I dont really understand this. Is it related to kv-parser or parsers in general?</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div></div> - delete a leading part of the message (skip until a specific character, for example), though now that I think of it, this could be probably done with a well-aimed scl that uses the csv-parser.<br></div></div></div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">Hmmm is this something like the marker() option for json-parser()?</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div>Regards,<br><br></div>Robert<br></div><div class="elided-text"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 12, 2016 at 5:09 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">In Pr #1225 I have added one more option to kv-parser(), extract-stray-words() that would put things the parser didnt recognize in a separate name value pair.</div><div dir="auto"><br></div><div dir="auto"><br></div></div><div><div class="m_5096337728938760499h5"><div class="gmail_extra"><br><div class="gmail_quote">On Dec 11, 2016 22:31, "David Boles (dboles)" <<a href="mailto:dboles@micron.com" target="_blank">dboles@micron.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Great - thank you for confirming what I thought I was seeing. Having '.' in the allowed range would be great but I can work around the issue. Another question - from the Linux kernel my log messages have the following form (from /dev/kmesg):<br>
<br>
<br>
3,2402,677449975886,-;[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12<br>
mse_space=kernel<br>
mse_version=1.0.0-r0.20161210<wbr>.f9b5a91.dirty<br>
mse_branch=frog<br>
mse_0_category=mse_error<br>
mse_0_version=0<br>
mse_0_code=12<br>
mse_0_file=foo.c<br>
mse_0_line=23<br>
mse_0_description=errno 12<br>
<br>
The kv-parser works great to obtain the key/value pairs from the above and $(format-json ...) lets me output them just fine. The next thing I face is that I want the "human message" part as well but when I refer to $MESSAGE and friends they always have the whole message. I believe the $(python ...) mechanism will allow me to snip out what I want, which is:<br>
<br>
[MSE] test log message in /home/dboles/...: Error@foo.c:23: errno 12<br>
<br>
I was curious if there was a better pattern or approach.<br>
<br>
Thanks,<br>
<br>
- db<br>
<br>
<br>
<br>
______________________________<wbr>__<br>
From: syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balab<wbr>it.hu</a>> on behalf of Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>><br>
Sent: Sunday, December 11, 2016 12:13 AM<br>
To: Syslog-ng users' and developers' mailing list<br>
Subject: Re: [syslog-ng] character handling in kv-parser<br>
<br>
Keys have to consist of [A-Za-z0-9-_] and doing a restriction is pretty important to identify keys in odd formats.<br>
<br>
We can change the set of characters pretty easily but for now it is hardcoded.<br>
<br>
I have a pending patchset that adds '.' as a char, as I needed that, and that's what syslog-ng uses to separate key portions.<br>
<br>
Would using a dot work for you? Then it would ve pretty easy to do the json conversion right within syslog-ng.<br>
<br>
<br>
Something like:<br>
<br>
parser { kv-parser(); };<br>
destination { file("foobar" template("$(format-json *)\n")); };<br>
<br>
Format json would generate embedded objects at dot boundaries.<br>
<br>
On Dec 11, 2016 00:03, "David Boles (dboles)" <<a href="mailto:dboles@micron.com" target="_blank">dboles@micron.com</a><mailto:<a href="mailto:dboles@micron.com" target="_blank">dbol<wbr>es@micron.com</a>>> wrote:<br>
I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal".<br>
<br>
When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings.<br>
<br>
Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact?<br>
<br>
Thanks,<br>
<br>
- db<br>
<br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
</blockquote></div></div>
</div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
</div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>