[syslog-ng] match filter scale issue

[Kavita Salve]

Hello,

                I have an issue related to the syslog-ng “match” filter on
which I need some inputs here.



                I have 800 “match” filters used in syslog configuration to
filter syslogs with priority 3, 4, 5 and 6 based on the content of the
syslog. So any incoming syslog’s will get forwarded only if the match is
found in those 800 filters. I am using the match filter with the macro
value(MSGHDR)
and value(MSG) depending on where I need to search the given regex for the
syslog.

E.g.

·         match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match
in header

·         match (“STANDBY: kernel” value(MSG)) - tries to find match in
message body (this is same as using message filter)



The filters work perfectly fine, but we see that there is a severe
performance degradation even when we use the match filter with macros -
value(MSGHDR) and value(MSG). From documentation, I see that we are not
supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that
matches both header and message) due to performance issues, but we should
be able to use the match filter with macros.



We did a scale test to find the eps that is getting processed by the
syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU
and 80 GB hard disk.



Following are the scale test results we are seeing for various combinations
of message and match filters:

1.       When absolutely no filters are used (i.e. no match /message
filters used, its only plain syslog forwarding everything from source to
destination), we got eps for processing syslogs around *6667 eps.*

2.       When I used ‘match’ filter (plain match filter without macros), it
resulted in eps of processing syslogs being around *1150-1200.*

3.       When I used message filter, it resulted in eps of processing
syslogs as *4730 eps*

4.       When I used ‘match’ filter with value’ (macro value(MSG) and
value(MSGHDR) used), it resulted in eps of *~1080eps*



*My analysis and questions:*

1.       I am seeing that even if I use one match filter in syslog-ng
configuration, the eps is coming down drastically. This happens even when I
use the match filter with macros. *How do I solve this issue? Is this
expected? Is there anything wrong in the way I am using the filters?*

2.       I cannot use the alternative “message” filter as I need to match
syslogs based on the MSGHDR also in many cases and “message” filter cannot
do this*. Is there any other way/filter to use here?*

*3.       *I want to find out why match filter with value macro is also
causing performance issues. Syslog-ng gives warning if I use match without
value that this will lead to low performance, but even with using value
macro I am seeing the performance hit*. I want help on how to solve this
issue. Can we get comparable eps with match filter where compared to
message filter?*



*PS:*

I am filtering syslogs generated by Cisco IOS devices that are in the
format:



*%FACILITY-SEVERITY-MNEMONIC: Message-text*

E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Loopback0, changed state to up



For these kind of syslogs, we are seeing that syslog-ng treats
*“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “*Message-text*”
as the *Message Body*.


Thanks

Kavita
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180114/c0350b47/attachment.html>