<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 8:26 AM, Nagy, Gábor <span dir="ltr"><<a href="mailto:gabor.nagy@balabit.com" target="_blank">gabor.nagy@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Asif,<div><br></div><div>If you need add-contextual-data can you upgrade using one of our unofficial binaries maybe?<br><a href="https://syslog-ng.com/blog/installing-latest-syslog-ng-on-rhel-and-other-rpm-distributions/#_ga=2.60852911.24278624.1521459042-804758321.1501593964" target="_blank">https://syslog-ng.com/blog/ins<wbr>talling-latest-syslog-ng-on-<wbr>rhel-and-other-rpm-distributio<wbr>ns/#_ga=2.60852911.24278624.<wbr>1521459042-804758321.150159396<wbr>4</a><br></div><div><br></div><div>Can you describe your use case with examples, please?</div></div></blockquote><div><br></div><div><br></div><div>My client hostname is svl-search-01 and its IP resolves to svl-remote-01. Its syslogs do not have any PRI or hostname in HOST field.</div><div><br></div><div>I like to have svl-search-01 in the HOST field.</div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Even if you set keep-hostname to 'yes' and message doesn't contain a hostname, syslog-ng will add a hostname/IP to the message depending on use-dns() option.</div><div>See `keep-hostname` option in doc:</div><div><a href="https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-options.html" target="_blank">https://syslog-ng.com/<wbr>documents/html/syslog-ng-ose-<wbr>latest-guides/en/syslog-ng-<wbr>ose-guide-admin/html/<wbr>reference-options.html</a><br></div><div><br></div><div>Regards,</div><div>Gabor</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 12:48 AM, Asif Iqbal <span dir="ltr"><<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Tue, Mar 20, 2018 at 10:23 AM, Asif Iqbal <span dir="ltr"><<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_4961851621391923992m_-8737793536210475710gmail-">On Tue, Mar 20, 2018 at 7:21 AM, Nagy, Gábor <span dir="ltr"><<a href="mailto:gabor.nagy@balabit.com" target="_blank">gabor.nagy@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Asif!<br><br>I think the problem with the first message comes from the message structure and the filter statement.<div>Its structure does not conform to either syslog RFC standards (RFC3164 or RFC5424).</div><div>Syslog-ng still tries to parse the log message by its internal heuristics and the "alarmLog" text is parsed as the 'program' field.</div><div>You can debug syslog-ng parsing with the format-json template in the destination:</div><div> `template("$(format-json -s syslog-proto)\n")`</div><div><br></div><div>The output for this was:</div><div><div>{"PROGRAM":"alarmLog,","PRIORI<wbr>TY":"notice","MESSAGE":"applia<wbr>nceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2, alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568, alarmText=delay:9 msec, siteName=Branch-UC1","HOST":"+<wbr>0000","FACILITY":"user","DATE"<wbr>:"Mar 13 23:49:48"}</div></div><div><br></div><div>The filter statement uses the `match()` filter which works on both the header and message part of the log message and thus would match for the first log message if the `value("MESSAGE")` part would not be there.</div><div>With that you restricted the filter to match only on the message part.</div><div>If you remove the value("MESSAGE") from the filter statement it will work.<br></div></div></blockquote><div><br></div></span><div>That was it. It is working now!! </div></div></div></div></blockquote><div><br></div><div><br></div></span><div>With global option keep_hostname(no) and use_dns(yes), I get the host A record gets prepended to the log.</div><div><br></div><div>But I like to have the hostname of the client prepended instead, which is not same as the dns A name of the client</div><div><br></div><div>I have about 40 clients in this group</div><div><br></div><div>If I use keep_hostname(yes), it provides no hostname at all, probably because of these syslogs do not conform with syslog RFC standard?</div><div><br></div><div> source s_alarm { udp( port(514) keep_hostname(yes) ); };</div><div><span><div> destination d_alarm { file("/var/log/alarms.log"); };</div></span><div> filter f_alarm { match("alarmLog" value("PROGRAM")); };</div><div> log { source(s_alarm); filter(f_alarm); destination(d_alarm); };</div></div><div><br></div><div>faxmodem (from #syslog-ng) gave me few options like using local /etc/hosts for lookup or using a metadata[1] from external file. </div><div>That feature, looks like, introduced on version 3.8. I am running latest centos 7 with syslog-ng version 3.5.6.</div><div><br></div><div>Is there a easy way to prepend the actual hostname of the client?</div><div><br></div><div>Thanks for your help</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>[1] <a href="https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/data-enrichment-add-contextual-data.html" target="_blank">https://syslog-ng.com/docu<wbr>ments/html/syslog-ng-ose-lates<wbr>t-guides/en/syslog-ng-ose-<wbr>guide-admin/html/data-enrichme<wbr>nt-add-contextual-data.html</a></div><div><div class="m_4961851621391923992h5"><div><br></div><div> </div><div> </div><div> </div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>BTW, is there a way to generate a feed a pcap to some program on the terminal to generate a json formatted output like this short from modifying the syslog-ng config</div><div>for destination to a template like you are showing?</div><div><br></div><div>Thanks for your help and I see the logs being written to the file now!</div><div><div class="m_4961851621391923992m_-8737793536210475710gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>Regards,</div><div>Gabor</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_4961851621391923992m_-8737793536210475710gmail-m_-1458833786342329624h5">On Tue, Mar 20, 2018 at 1:52 AM, Asif Iqbal <span dir="ltr"><<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_4961851621391923992m_-8737793536210475710gmail-m_-1458833786342329624h5"><div dir="ltr"><div>syslog-ng is <b><font color="#ff0000">NOT</font></b> writing syslog like this to a file which has no <<b>PRI</b>></div><div><br></div><div>23:49:48.306587 IP 192.168.1.100.39567 > 192.168.100.100.514: [|syslog]</div><div>E.....@.>..U...g..........T.20<wbr>18-03-19T23:49:48+0000 alarmLog, applianceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2, alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568, alarmText="delay:9 msec", siteName=Branch-UC1</div><div>................</div><div><br></div><div><br></div><div>syslog-ng is writing syslog like this to a file <b>OK </b></div><div><br></div><div>23:50:26.930023 IP 192.168.1.100.55078 > 192.168.100.100.514: SYSLOG <a href="http://mail.info" target="_blank">mail.info</a>, length: 76</div><div>E..h.B@.>......g.....&...Tt.<2<wbr>2>Mar 19 23:50:26 SVL-remotehost-02 root: this is third test alarmLog................</div><div><br></div><div><br></div><div>Here is my syslog-ng config</div><div> source s_udp { udp(ip(0.0.0.0) port(514)); };<br></div><div><div> destination d_alarm { file("/var/log/alarms.log"); };</div><div> filter f_alarm { match("alarmLog" value("MESSAGE")); };<br></div><div> log { source(s_udp); filter(f_alarm); destination(d_alarm); };<br></div></div><div><br></div><div>I am using syslog-ng version 3.5.6 on centos 7</div><div><br></div><div>Any idea why syslog-ng is writing the first log event into a file?</div><div><br></div><div>Appreciate any help!</div><span class="m_4961851621391923992m_-8737793536210475710gmail-m_-1458833786342329624m_5195631471607390889HOEnZb"><font color="#888888"><div><br></div><div><br></div>-- <br><div class="m_4961851621391923992m_-8737793536210475710gmail-m_-1458833786342329624m_5195631471607390889m_-7794152780587771181gmail_signature">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
</font></span></div>
<br></div></div>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div></div></div><div><div class="m_4961851621391923992m_-8737793536210475710gmail-h5"><br><br clear="all"><div><br></div>-- <br><div class="m_4961851621391923992m_-8737793536210475710gmail-m_-1458833786342329624gmail_signature">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
</div></div></div></div>
</blockquote></div></div></div><div><div class="m_4961851621391923992h5"><br><br clear="all"><div><br></div>-- <br><div class="m_4961851621391923992m_-8737793536210475710gmail_signature">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
</div></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
</div></div>