Hi, On Sun, Jan 14, 2018 at 6:10 PM, Kavita Salve <kavita.mohite@gmail.com> wrote:
Hello,
I have an issue related to the syslog-ng “match” filter on which I need some inputs here.
I have 800 “match” filters used in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on the content of the syslog. So any incoming syslog’s will get forwarded only if the match is found in those 800 filters. I am using the match filter with the macro value(MSGHDR) and value(MSG) depending on where I need to search the given regex for the syslog.
E.g.
· match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match in header
· match (“STANDBY: kernel” value(MSG)) - tries to find match in message body (this is same as using message filter)
The filters work perfectly fine, but we see that there is a severe performance degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG). From documentation, I see that we are not supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that matches both header and message) due to performance issues, but we should be able to use the match filter with macros.
We did a scale test to find the eps that is getting processed by the syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk.
Following are the scale test results we are seeing for various combinations of message and match filters:
1. When absolutely no filters are used (i.e. no match /message filters used, its only plain syslog forwarding everything from source to destination), we got eps for processing syslogs around *6667 eps.*
This number is pretty low, syslog-ng should be able to cope with over 100k per second without filters into local files. Are you sending it somewhere else? What CPU are you using?
2. When I used ‘match’ filter (plain match filter without macros), it resulted in eps of processing syslogs being around *1150-1200.*
3. When I used message filter, it resulted in eps of processing syslogs as *4730 eps*
4. When I used ‘match’ filter with value’ (macro value(MSG) and value(MSGHDR) used), it resulted in eps of *~1080eps*
- internally match(value(MSG)) should be the same as message(). And a single match(value(MSG)) and match(value(MSGHDR)) should roughly perform the same. - the behaviour presented by match() without value() used to be the default behaviour before 3.0, where there was no value() parameter - match() without the value() parameter is emulated by formatting the string "$MSGHDR$MSG" using printf() and matching against that. It is reported to be slower because of this printf() and allocation logic. - It's strange that for you this emulation logic is faster than combining two separate filters - Obviously if you combine filters, that would decrease performance again. Can you please report the version of syslog-ng you are using? I see these potential actions in the order of easy to more difficult: - you could use pcre based regexps, latest versions have JIT in regexp evaluation - you could perhaps speed up the regexps themselves. they are inherently slow. match tries to locate the pattern in the entire string, which is slow if the input is long. making patterns more specific speeds things up. - you could use flags(no-parse) on input, which would speed up parsing, which is not handling the Cisco messages completely accurately anyway. This way you can parse against the entire line as received - the match() without value() could be made faster, but that's already a code change, by avoiding the use of printf() and malloc() I am not sure where that would put us - you could also use the cisco-parser() stuff in recent syslog-ng, that copes with Cisco logs better, but tries to be pretty accurate, so would be slower possibly I hope this helps,
*My analysis and questions:*
1. I am seeing that even if I use one match filter in syslog-ng configuration, the eps is coming down drastically. This happens even when I use the match filter with macros. *How do I solve this issue? Is this expected? Is there anything wrong in the way I am using the filters?*
2. I cannot use the alternative “message” filter as I need to match syslogs based on the MSGHDR also in many cases and “message” filter cannot do this*. Is there any other way/filter to use here?*
*3. *I want to find out why match filter with value macro is also causing performance issues. Syslog-ng gives warning if I use match without value that this will lead to low performance, but even with using value macro I am seeing the performance hit*. I want help on how to solve this issue. Can we get comparable eps with match filter where compared to message filter?*
*PS:*
I am filtering syslogs generated by Cisco IOS devices that are in the format:
*%FACILITY-SEVERITY-MNEMONIC: Message-text*
E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
For these kind of syslogs, we are seeing that syslog-ng treats *“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “ *Message-text*” as the *Message Body*.