Hi, Can it happen that syslog-ng becomes CPU bound when you enable the filter? Can you check that using top/vmstat/younameit whether syslog-ng is spinning on a CPU? the host() filter is using regexps, and although your host() pattern doesn't seem complex, it was anonimized in the config, so I don't know if it is more complex in reality. You might want to use a simple substring filter, like so: filter f_firewalls { host("ipaddress" type(string)); }; If you need more than mere substring matching, you might try to use "pcre" (if your syslog-ng is compiled with that), or "glob", which permits shell-like wildcard matches. Otherwise I have no clue why that filter reduces perfomance so much, it shouldn't, but I don't know how much stress you put to it. Hope this helps, Bazsi On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
Initially sent to the wrong address....
Begin forwarded message:
From: Paul Hutton <paul_hutton@bigpond.com>
Subject: Syslog NG simple filtering not working as expected...
Date: 26 September 2013 11:23:15 PM AEST
To: syslog-ng-request@lists.balabit.hu
Hi All,
I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to:
1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out)
In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity.
My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line
filter (not_firewalls);
in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic.
I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site.
I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there...
Thanks in advance.
Paul
@version: 3.2 @include "scl.conf" options { stats_level(3); flush_lines(100); flush_timeout(10000); }; ################################################################ # Ensure syslog-ng logs this host similarly to syslogd # ################################################################ source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # # Set global network source # source src { udp(ip("0.0.0.0") port (514)); };
############################################################### # Local logfiles and remote destinations setup # ############################################################### destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loginlog { file("/var/adm/loginlog"); }; # # Set remote destinations # destination Remote_L { udp ("10.octet.octet.octetL" port (514) ); }; destination Remote_D1 { udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); }; destination Remote_D2 { udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); }; destination Remote_C { udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); }; # # Set local destination for all remote syslog input # destination d_logger { file("/var/log/logger.log"); }; # # Add another local debug file to check filters are working # destination d_debug { file("/var/log/debug.log"); }; ############################################################## # Filters for mimicking Solaris syslogd # # identifying firewall syslogs # # identifying severity levels 0 to 2 # ############################################################## filter f_syslog { facility(mail) or (facility(daemon) and priority(notice)); }; filter f_messages { priority(err) or facility(kern) or (facility(user) and priority(err)) or (facility(daemon) and priority(notice)) or (facility(mail) and priority(crit)); }; filter f_auth { facility (auth); };
# # Set Firewall Filters # filter firewalls { host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); }; filter not_firewalls { not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); };
# # Filter Severity Levels 0, 1 and 2 # filter f_crit { level (crit .. emerg); }; ############################################################## # Now do some logging # ##############################################################
# # Log local syslog data appropriately # log { source(s_local); filter(f_syslog); destination(syslog); }; log { source(s_local); filter(f_messages); destination(messages); }; log { source(s_local); filter(f_auth); destination(loginlog); };
# # Log remotely sourced syslogs of all Severity Levels to remote L system # but exclude syslogs from the firewalls # log { source (src); filter (not_firewalls); destination (Remote_L); destination (d_debug); }; # # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems # including syslogs from the firewalls # #log { # source (src); # filter (f_crit); # destination (Remote_D1); # destination (Remote_D2); #}; # # Log remotely sourced syslogs from the two firewalls to remote C system # #log { #source (src); #filter (firewalls); # destination (Remote_C); #}; # # Log remotely sourced syslogs to a local file # log { source (src); destination (d_logger); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq