<div dir="ltr"><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Kavita Salve</b> <span dir="ltr"><<a href="mailto:kavita.mohite@gmail.com">kavita.mohite@gmail.com</a>></span><br>Date: Sun, Jan 14, 2018 at 10:40 PM<br>Subject: match filter scale issue<br>To: Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br><br><br><div dir="ltr">
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">Hello,<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> I have an issue related to the
syslog-ng “match” filter on which I need some inputs here.<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> I have 800 “match” filters used
in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on
the content of the syslog. So any incoming syslog’s will get forwarded only if
the match is found in those 800 filters. I am using the match filter with the macro
value</span><span style="font-size:11pt">(MSGHDR)
and value(MSG) depending on where I need to search the given regex for the
syslog.<span></span></span></p>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">E.g. <span></span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt;font-family:Symbol">·<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">match (“<span style="color:rgb(51,51,51)">%LINEPROTO-5-UPDOWN</span>”
value(MSGHDR)) – tries to find match in header<span></span></span></p>
<span></span>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt;font-family:Symbol">·<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">match (“<span style="color:rgb(51,51,51)">STANDBY: kernel</span>”
value(MSG)) - tries to find match in message body (this is same as using
message filter)<span></span></span></p>
<span></span>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">The
filters work perfectly fine, but we see that there is a severe performance
degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG).
>From documentation, I see that we are not supposed to use plain match filter (E.g.
match (“<span style="color:rgb(51,51,51)">%LINEPROTO-5-UPDOWN</span>”) that matches
both header and message) due to performance issues, but we should be able to
use the match filter with macros.<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">We
did a scale test to find the eps that is getting processed by the syslog-ng
server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk. <span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">Following
are the scale test results we are seeing for various combinations of message
and match filters:<span></span></span></p>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">1.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When absolutely no filters are used (i.e. no
match /message filters used, its only plain syslog forwarding everything from
source to destination), we got eps for processing syslogs around <b>6667
eps.</b></span><span style="font-size:11pt"><span></span></span></p>
<span></span>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">2.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When I used ‘match’ filter (plain match filter
without macros), it resulted in eps of processing syslogs being around <b>1150-1200<span></span>.</b></span><span style="font-size:11pt"><span></span></span></p>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">3.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When I used message filter, it resulted in eps
of processing syslogs as <b>4730 eps</b></span><span style="font-size:11pt"><span></span></span></p>
<span></span>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">4.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt;color:black">When I used ‘match’ filter with value’ (macro </span><span style="font-size:11pt">value(MSG) and
value(MSGHDR) used)<span style="color:black">, it resulted in eps of <b>~1080eps</b></span><span></span></span></p>
<span></span>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt">My analysis
and questions:<span></span></span></b></p>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">1.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">I am seeing that even if I use one match filter in syslog-ng
configuration, the eps is coming down drastically. This happens even when I use
the match filter with macros. <b>How do I solve
this issue? Is this expected? Is there anything wrong in the way I am using the
filters?</b><span></span></span></p>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">2.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span><span style="font-size:11pt">I cannot use the alternative “message” filter as I need to
match syslogs based on the MSGHDR also in many cases and “message” filter cannot
do this<b>. Is there any other way/filter
to use here?</b><span></span></span></p>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 1in;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt">3.<span style="font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman""> </span></span></b><span style="font-size:11pt">I want to
find out why match filter with value macro is also causing performance issues.
Syslog-ng gives warning if I use match without value that this will lead to low
performance, but even with using value macro I am seeing the performance hit<b>. I want help on how to solve this issue.<span></span>
Can we get comparable eps with match filter where compared to message filter?<span></span></b></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt"><span> </span></span></b></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt">PS:<span></span></span></b></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">I
am filtering syslogs generated by Cisco IOS devices that are in the format:<span></span></span></p>
<span></span>
<p class="m_-5921160889916049877gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt 0.5in;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> </span><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><b><span style="font-size:11pt;color:rgb(51,51,51)">%FACILITY-SEVERITY-MNEMONIC:
Message-text</span></b><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt;color:rgb(51,51,51)">E.g.: *Mar 6
22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0,
changed state to up</span><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"> </span><span></span><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt">For
these kind of syslogs, we are seeing that syslog-ng treats <b>“<span style="color:rgb(51,51,51)">%FACILITY-SEVERITY-MNEMONIC:</span></b><span style="color:rgb(51,51,51)">” as the <b>Message Header</b> and “<b>Message-text</b>”
as the <b>Message Body</b>. </span></span><span style="font-size:11pt"><span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span style="color:rgb(51,51,51)"><br></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span style="color:rgb(51,51,51)">Thanks</span></span></p><span class="HOEnZb"><font color="#888888"><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span style="color:rgb(51,51,51)">Kavita</span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri"><span style="font-size:11pt"><span> </span></span></p>
</font></span></div>
</div><br></div>