[syslog-ng] match filter scale issue

[Scheidler, Balázs]

Hi,


On Sun, Jan 14, 2018 at 6:10 PM, Kavita Salve <kavita.mohite at gmail.com>
wrote:

> Hello,
>
>                 I have an issue related to the syslog-ng “match” filter on
> which I need some inputs here.
>
>
>
>                 I have 800 “match” filters used in syslog configuration to
> filter syslogs with priority 3, 4, 5 and 6 based on the content of the
> syslog. So any incoming syslog’s will get forwarded only if the match is
> found in those 800 filters. I am using the match filter with the macro value(MSGHDR)
> and value(MSG) depending on where I need to search the given regex for the
> syslog.
>
> E.g.
>
> ·         match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find
> match in header
>
> ·         match (“STANDBY: kernel” value(MSG)) - tries to find match in
> message body (this is same as using message filter)
>
>
>
> The filters work perfectly fine, but we see that there is a severe
> performance degradation even when we use the match filter with macros -
> value(MSGHDR) and value(MSG). From documentation, I see that we are not
> supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”)
> that matches both header and message) due to performance issues, but we
> should be able to use the match filter with macros.
>
>
>
> We did a scale test to find the eps that is getting processed by the
> syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU
> and 80 GB hard disk.
>
>
>
> Following are the scale test results we are seeing for various
> combinations of message and match filters:
>
> 1.       When absolutely no filters are used (i.e. no match /message
> filters used, its only plain syslog forwarding everything from source to
> destination), we got eps for processing syslogs around *6667 eps.*
>

This number is pretty low, syslog-ng should be able to cope with over 100k
per second without filters into local files. Are you sending it somewhere
else? What CPU are you using?


> 2.       When I used ‘match’ filter (plain match filter without macros),
> it resulted in eps of processing syslogs being around *1150-1200.*
>
> 3.       When I used message filter, it resulted in eps of processing
> syslogs as *4730 eps*
>
> 4.       When I used ‘match’ filter with value’ (macro value(MSG) and
> value(MSGHDR) used), it resulted in eps of *~1080eps*
>


   - internally match(value(MSG)) should be the same as message(). And a
   single match(value(MSG)) and match(value(MSGHDR)) should roughly perform
   the same.
   - the behaviour presented by match() without value() used to be the
   default behaviour before 3.0, where there was no value() parameter
   - match() without the value() parameter is emulated by formatting the
   string "$MSGHDR$MSG" using printf() and matching against that. It is
   reported to be slower because of this printf() and allocation logic.
   - It's strange that for you this emulation logic is faster than
   combining two separate filters
   - Obviously if you combine  filters, that would decrease performance
   again.

Can you please report the version of syslog-ng you are using?

I see these potential actions in the order of easy to more difficult:

   - you could use pcre based regexps, latest versions have JIT in regexp
   evaluation
   - you could perhaps speed up the regexps themselves. they are inherently
   slow. match tries to locate the pattern in the entire string, which is slow
   if the input is long. making patterns more specific speeds things up.
   - you could use flags(no-parse) on input, which would speed up parsing,
   which is not handling the Cisco messages completely accurately anyway. This
   way you can parse against the entire line as received
   - the match() without value() could be made faster, but that's already a
   code change, by avoiding the use of printf() and malloc() I am not sure
   where that would put us
   - you could also use the cisco-parser() stuff in recent syslog-ng, that
   copes with Cisco logs better, but tries to be pretty accurate, so would be
   slower possibly

I hope this helps,




>
>
> *My analysis and questions:*
>
> 1.       I am seeing that even if I use one match filter in syslog-ng
> configuration, the eps is coming down drastically. This happens even when I
> use the match filter with macros. *How do I solve this issue? Is this
> expected? Is there anything wrong in the way I am using the filters?*
>
> 2.       I cannot use the alternative “message” filter as I need to match
> syslogs based on the MSGHDR also in many cases and “message” filter cannot
> do this*. Is there any other way/filter to use here?*
>
> *3.       *I want to find out why match filter with value macro is also
> causing performance issues. Syslog-ng gives warning if I use match without
> value that this will lead to low performance, but even with using value
> macro I am seeing the performance hit*. I want help on how to solve this
> issue. Can we get comparable eps with match filter where compared to
> message filter?*
>
>
>
> *PS:*
>
> I am filtering syslogs generated by Cisco IOS devices that are in the
> format:
>
>
>
> *%FACILITY-SEVERITY-MNEMONIC: Message-text*
>
> E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on
> Interface Loopback0, changed state to up
>
>
>
> For these kind of syslogs, we are seeing that syslog-ng treats
> *“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “
> *Message-text*” as the *Message Body*.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180114/f501a6e8/attachment.html>