Hello,

                I have an issue related to the syslog-ng “match” filter on which I need some inputs here.

 

                I have 800 “match” filters used in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on the content of the syslog. So any incoming syslog’s will get forwarded only if the match is found in those 800 filters. I am using the match filter with the macro value(MSGHDR) and value(MSG) depending on where I need to search the given regex for the syslog.

E.g.

·         match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match in header

·         match (“STANDBY: kernel” value(MSG)) - tries to find match in message body (this is same as using message filter)

 

The filters work perfectly fine, but we see that there is a severe performance degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG). >From documentation, I see that we are not supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that matches both header and message) due to performance issues, but we should be able to use the match filter with macros.

 

We did a scale test to find the eps that is getting processed by the syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk.

 

Following are the scale test results we are seeing for various combinations of message and match filters:

1.       When absolutely no filters are used (i.e. no match /message filters used, its only plain syslog forwarding everything from source to destination), we got eps for processing syslogs around 6667 eps.

2.       When I used ‘match’ filter (plain match filter without macros), it resulted in eps of processing syslogs being around 1150-1200.

3.       When I used message filter, it resulted in eps of processing syslogs as 4730 eps

4.       When I used ‘match’ filter with value’ (macro value(MSG) and value(MSGHDR) used), it resulted in eps of ~1080eps

 

My analysis and questions:

1.       I am seeing that even if I use one match filter in syslog-ng configuration, the eps is coming down drastically. This happens even when I use the match filter with macros. How do I solve this issue? Is this expected? Is there anything wrong in the way I am using the filters?

2.       I cannot use the alternative “message” filter as I need to match syslogs based on the MSGHDR also in many cases and “message” filter cannot do this. Is there any other way/filter to use here?

3.       I want to find out why match filter with value macro is also causing performance issues. Syslog-ng gives warning if I use match without value that this will lead to low performance, but even with using value macro I am seeing the performance hit. I want help on how to solve this issue. Can we get comparable eps with match filter where compared to message filter?

 

PS:

I am filtering syslogs generated by Cisco IOS devices that are in the format:

 

%FACILITY-SEVERITY-MNEMONIC: Message-text

E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up

 

For these kind of syslogs, we are seeing that syslog-ng treats “%FACILITY-SEVERITY-MNEMONIC:” as the Message Header and “Message-text” as the Message Body.

Thanks

Kavita