<div dir="ltr">Hi,<br><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 14, 2018 at 6:10 PM, Kavita Salve <span dir="ltr"><<a href="mailto:kavita.mohite@gmail.com" target="_blank">kavita.mohite@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">Hello,<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> I have an issue related to the
syslog-ng “match” filter on which I need some inputs here.<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> I have 800 “match” filters used
in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on
the content of the syslog. So any incoming syslog’s will get forwarded only if
the match is found in those 800 filters. I am using the match filter with the macro
value</span><span style="font-size:11pt">(MSGHDR)
and value(MSG) depending on where I need to search the given regex for the
syslog.<span></span></span></p>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">E.g. <span></span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt;font-family:Symbol">·<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">match (“<span style="color:rgb(51,51,51)">%LINEPROTO-5-UPDOWN</span>”
value(MSGHDR)) – tries to find match in header<span></span></span></p>
<span></span>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt;font-family:Symbol">·<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">match (“<span style="color:rgb(51,51,51)">STANDBY: kernel</span>”
value(MSG)) - tries to find match in message body (this is same as using
message filter)<span></span></span></p>
<span></span>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">The
filters work perfectly fine, but we see that there is a severe performance
degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG).
>From documentation, I see that we are not supposed to use plain match filter (E.g.
match (“<span style="color:rgb(51,51,51)">%LINEPROTO-5-UPDOWN</span>”) that matches
both header and message) due to performance issues, but we should be able to
use the match filter with macros.<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">We
did a scale test to find the eps that is getting processed by the syslog-ng
server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk. <span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">Following
are the scale test results we are seeing for various combinations of message
and match filters:<span></span></span></p>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">1.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When absolutely no filters are used (i.e. no
match /message filters used, its only plain syslog forwarding everything from
source to destination), we got eps for processing syslogs around <b>6667
eps.</b></span><span style="font-size:11pt"><span></span></span></p></div></blockquote><div><br></div><div>This number is pretty low, syslog-ng should be able to cope with over 100k per second without filters into local files. Are you sending it somewhere else? What CPU are you using?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"></p>
<span></span>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">2.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When I used ‘match’ filter (plain match filter
without macros), it resulted in eps of processing syslogs being around <b>1150-1200<span></span>.</b></span><span style="font-size:11pt"><span></span></span></p>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">3.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When I used message filter, it resulted in eps
of processing syslogs as <b>4730 eps</b></span><span style="font-size:11pt"><span></span></span></p>
<span></span>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">4.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When I used ‘match’ filter with value’ (macro </span><span style="font-size:11pt">value(MSG) and
value(MSGHDR) used)<span style="color:black">, it resulted in eps of <b>~1080eps</b></span></span></p></div></blockquote><div><br></div><div><ul><li>internally match(value(MSG)) should be the same as message(). And a single match(value(MSG)) and match(value(MSGHDR)) should roughly perform the same. </li><li>the behaviour presented by match() without value() used to be the default behaviour before 3.0, where there was no value() parameter</li><li>match() without the value() parameter is emulated by formatting the string "$MSGHDR$MSG" using printf() and matching against that. It is reported to be slower because of this printf() and allocation logic.</li><li>It's strange that for you this emulation logic is faster than combining two separate filters</li><li>Obviously if you combine filters, that would decrease performance again.</li></ul></div><div>Can you please report the version of syslog-ng you are using?</div><div><br></div><div>I see these potential actions in the order of easy to more difficult:</div><div><ul><li>you could use pcre based regexps, latest versions have JIT in regexp evaluation</li><li>you could perhaps speed up the regexps themselves. they are inherently slow. match tries to locate the pattern in the entire string, which is slow if the input is long. making patterns more specific speeds things up.<br></li><li>you could use flags(no-parse) on input, which would speed up
parsing, which is not handling the Cisco messages completely accurately
anyway. This way you can parse against the entire line as received</li><li>the match() without value() could be made faster, but that's already a code change, by avoiding the use
of printf() and malloc() I am not sure where that would put us</li><li>you could also use the cisco-parser() stuff in recent syslog-ng, that copes with Cisco logs better, but tries to be pretty accurate, so would be slower possibly<br></li></ul><p>I hope this helps,<br></p><p> </p></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span></span></span></p>
<span></span>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> <br></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt">My analysis
and questions:<span></span></span></b></p>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">1.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">I am seeing that even if I use one match filter in syslog-ng
configuration, the eps is coming down drastically. This happens even when I use
the match filter with macros. <b>How do I solve
this issue? Is this expected? Is there anything wrong in the way I am using the
filters?</b><span></span></span></p>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">2.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">I cannot use the alternative “message” filter as I need to
match syslogs based on the MSGHDR also in many cases and “message” filter cannot
do this<b>. Is there any other way/filter
to use here?</b><span></span></span></p>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt">3.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman""> </span></span></b><span style="font-size:11pt">I want to
find out why match filter with value macro is also causing performance issues.
Syslog-ng gives warning if I use match without value that this will lead to low
performance, but even with using value macro I am seeing the performance hit<b>. I want help on how to solve this issue.<span></span>
Can we get comparable eps with match filter where compared to message filter?<span></span></b></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt"><span> </span></span></b></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt">PS:<span></span></span></b></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">I
am filtering syslogs generated by Cisco IOS devices that are in the format:<span></span></span></p>
<span></span>
<p class="gmail-m_-4065865945168475543gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> </span><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt;color:rgb(51,51,51)">%FACILITY-SEVERITY-MNEMONIC:
Message-text</span></b><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt;color:rgb(51,51,51)">E.g.: *Mar 6
22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0,
changed state to up</span><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> </span><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">For
these kind of syslogs, we are seeing that syslog-ng treats <b>“<span style="color:rgb(51,51,51)">%FACILITY-SEVERITY-MNEMONIC:</span></b><span style="color:rgb(51,51,51)">” as the <b>Message Header</b> and “<b>Message-text</b>”
as the <b>Message Body</b>. </span></span><span style="font-size:11pt"><span></span></span></p><br></div></blockquote><div><br></div><div><br></div><div> </div></div></div></div>