---------- Forwarded message ---------- From: Kavita Salve <kavita.mohite@gmail.com> Date: Sun, Jan 14, 2018 at 10:40 PM Subject: match filter scale issue To: Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> Hello, I have an issue related to the syslog-ng “match” filter on which I need some inputs here. I have 800 “match” filters used in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on the content of the syslog. So any incoming syslog’s will get forwarded only if the match is found in those 800 filters. I am using the match filter with the macro value(MSGHDR) and value(MSG) depending on where I need to search the given regex for the syslog. E.g. · match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match in header · match (“STANDBY: kernel” value(MSG)) - tries to find match in message body (this is same as using message filter) The filters work perfectly fine, but we see that there is a severe performance degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG). From documentation, I see that we are not supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that matches both header and message) due to performance issues, but we should be able to use the match filter with macros. We did a scale test to find the eps that is getting processed by the syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk. Following are the scale test results we are seeing for various combinations of message and match filters: 1. When absolutely no filters are used (i.e. no match /message filters used, its only plain syslog forwarding everything from source to destination), we got eps for processing syslogs around *6667 eps.* 2. When I used ‘match’ filter (plain match filter without macros), it resulted in eps of processing syslogs being around *1150-1200.* 3. When I used message filter, it resulted in eps of processing syslogs as *4730 eps* 4. When I used ‘match’ filter with value’ (macro value(MSG) and value(MSGHDR) used), it resulted in eps of *~1080eps* *My analysis and questions:* 1. I am seeing that even if I use one match filter in syslog-ng configuration, the eps is coming down drastically. This happens even when I use the match filter with macros. *How do I solve this issue? Is this expected? Is there anything wrong in the way I am using the filters?* 2. I cannot use the alternative “message” filter as I need to match syslogs based on the MSGHDR also in many cases and “message” filter cannot do this*. Is there any other way/filter to use here?* *3. *I want to find out why match filter with value macro is also causing performance issues. Syslog-ng gives warning if I use match without value that this will lead to low performance, but even with using value macro I am seeing the performance hit*. I want help on how to solve this issue. Can we get comparable eps with match filter where compared to message filter?* *PS:* I am filtering syslogs generated by Cisco IOS devices that are in the format: *%FACILITY-SEVERITY-MNEMONIC: Message-text* E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up For these kind of syslogs, we are seeing that syslog-ng treats *“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “*Message-text*” as the *Message Body*. Thanks Kavita