<div dir="auto"><div>Hi,<div dir="auto"><br></div><div dir="auto"><br></div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 29, 2020, 19:10 Raghunath Adhyapak <<a href="mailto:funduraghu@gmail.com">funduraghu@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>I was planning to do the following</div><div>1. Receive events</div><div>2. Filter events matching format <134>1 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 Mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53</div><div>3. Rewrite epoch timestamp to isodate format</div><div>4. Use syslog-parser with flags(syslog-protocol)</div><div>5. Finally, write to destination</div><div></div></div></div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">This sounds good in principle. Try to make the first filter as specific as possible, to improve performance.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div><div>source s_syslog {</div><div> syslog(</div><div> transport(udp)</div><div> port(514)</div><div> );</div><div>};</div><div><br></div><div>filter f_version_and_epoch {</div><div> match ("(?:<[0-9]+>)(?:[0-9]+ )?(?:[0-9]+)(?:\.[0-9]+)? " value ("MESSAGE"));</div><div>};</div><div>rewrite epoch_to_isodate {</div><div> subst ("(?:<[0-9]+>)(?:[0-9]+ )?(?<seconds>[0-9]+)(?:\.[0-9]+)? ", "$(date --iso-8601=seconds -d @$seconds)", value("MESSAGE"));</div><div>};</div></div></div></div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">$(date) doesn't exist, the template language of syslog-ng is not the same as the shell language, so you can't freely use shell commands.</div><div dir="auto"><br></div><div dir="auto">To use date related features of syslog-ng, you could parse the timestamp using date-parser(), that can parse this format using %s (if i remember correctly, supports strptime formats).</div><div dir="auto"><br></div><div dir="auto">Then you can simply use any of syslog-ng's date/time macros.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div><div>log {</div><div> source(s_syslog);</div><div> filter(f_version_and_epoch);</div><div> rewrite(epoch_to_isodate);</div><div> parser {</div><div> syslog-parser(flags(syslog-protocol));</div><div> }</div><div> destination(d_file);</div><div>};</div></div><div><br></div><div>Let me know if this will work.</div><div><br></div><div>Currently, I'm getting errors while trying to convert timestamp in seconds to isodate.</div><div>Let me know if there is a better way.</div><div><br></div><div>Thanks</div><div>Raghu</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 29, 2020 at 12:38 AM Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" target="_blank" rel="noreferrer">bazsi77@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="auto">Rant mode on.<div dir="auto"><br></div><div dir="auto">This sucks. RFC5424 finally attempted to make an orderly protocol, with prettty explicit format requirements, to fix the shortcomings of rfc3164. What do vendors do? They explicitly specify that they are using rfc5424 and then ignore the very format.</div><div dir="auto"><br></div><div dir="auto">Why do they add "1 "? If they left that out, syslog-ng would parse this properly. With that we kick in strict syntax checking, exactly to avoid deteriorating the syntax of rfc5424, exactly what happened to syslog prior to that. History is repeating itself.</div><div dir="auto"><br></div><div dir="auto">Rant mode off.</div><div dir="auto"><br></div><div dir="auto">I might look into attempting rfc3164 if rfc5424 fails that would possibly solve this. I can't remember if we allow unix timestamp as an incoming format though.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 28, 2020, 17:52 Raghunath Adhyapak <<a href="mailto:funduraghu@gmail.com" target="_blank" rel="noreferrer">funduraghu@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>This log line was received from Cisco device.</div><div>I believe that the timestamp is not in acceptable format.</div><div>Is there any way we can configure syslog-ng to accept timestamp of this form?</div><div><br></div><div>Thanks</div><div>Raghu</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 28, 2020 at 6:26 PM Nagy Gábor <<a href="mailto:gabor.hl@gmail.com" rel="noreferrer noreferrer" target="_blank">gabor.hl@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi!<br><br></div><div>At first look it seems to me that your log message is not in RFC5424 [1] format, or in RFC3164 [2] format.<br></div><div>You use syslog() source driver which expects these formats.<br><br></div><div>Do you receive log messages from other clients too (I guess if you have max-connections(500))?<br></div><div>What device is the log source where messages are coming form?<br></div><div><br></div><div>Regards,</div><div>Gabor<br></div><div><br></div><div>[1] <a href="https://tools.ietf.org/html/rfc5424" rel="noreferrer noreferrer" target="_blank">https://tools.ietf.org/html/rfc5424</a><br>[2] <a href="https://tools.ietf.org/html/rfc3164" rel="noreferrer noreferrer" target="_blank">https://tools.ietf.org/html/rfc3164</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Raghunath Adhyapak <<a href="mailto:funduraghu@gmail.com" rel="noreferrer noreferrer" target="_blank">funduraghu@gmail.com</a>> ezt írta (időpont: 2020. ápr. 28., K, 13:00):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hello all,</div><div><br></div><div>I am receiving the following syslog line from one of devices.</div><div><br></div><div dir="ltr"><134>1 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 Mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53<br></div><div dir="ltr"><br></div><div>This line contains a version filed immediately following the priority, and then timestamp is in epoch format as against ISO8601 or other standard format.</div><div><br></div><div>I see the following error in syslog-ng log:</div><div>[2020-04-28T10:46:15.340911] Outgoing message; message='Apr 28 10:46:15 ip-172-31-240-95 syslog-ng[27873]: Error processing log message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53'<br></div><div><br></div><div>What could be the possible issue here?</div><div><br></div><div>My config is as follows:</div><div><br></div><div><div>##========================================</div><div>########################</div><div># Global options</div><div>########################</div><div>options {keep_hostname (yes); use_dns (no); mark-freq(30);};</div><div>########################</div><div># Sources</div><div>########################</div><div>source s_syslog {</div><div> syslog(</div><div> transport(udp)</div><div> port(514)</div><div> max-connections(500)</div><div> );</div><div>};</div></div><div><div>########################</div><div># Destinations</div><div>########################</div><div>destination d_file {<br></div><div> file("/var/log/dump.log");</div><div>};</div></div><div><div>########################</div><div># Log paths</div><div>########################</div><div>log {</div><div> source(s_syslog);</div><div> destination(d_file);</div><div> flags(flow-control);</div><div>};</div><div>##========================================</div></div><div><br></div><div>If I check my file /var/log/dump.log, I see that the error line is getting written to it too.</div><div><br></div><div><div>root@ip-172-31-240-95:~# tail -f /var/log/dump.log | grep "Error processing"</div><div>Apr 28 10:44:46 ip-172-31-240-95 syslog-ng[27873]: Error processing log message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53</div><div>Apr 28 10:46:15 ip-172-31-240-95 syslog-ng[27873]: Error processing log message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53</div></div><div><br></div><div>Thanks</div><div>Raghu</div></div></div></div></div></div></div></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div></div></div>