Fwd: Syslog NG simple filtering not working as expected...
Initially sent to the wrong address.... Begin forwarded message:
From: Paul Hutton <paul_hutton@bigpond.com> Subject: Syslog NG simple filtering not working as expected... Date: 26 September 2013 11:23:15 PM AEST To: syslog-ng-request@lists.balabit.hu
Hi All,
I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to:
1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out)
In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity.
My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line
filter (not_firewalls);
in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic.
I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site.
I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there...
Thanks in advance.
Paul
@version: 3.2 @include "scl.conf" options { stats_level(3); flush_lines(100); flush_timeout(10000); }; ################################################################ # Ensure syslog-ng logs this host similarly to syslogd # ################################################################ source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # # Set global network source # source src { udp(ip("0.0.0.0") port (514)); };
############################################################### # Local logfiles and remote destinations setup # ############################################################### destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loginlog { file("/var/adm/loginlog"); }; # # Set remote destinations # destination Remote_L { udp ("10.octet.octet.octetL" port (514) ); }; destination Remote_D1 { udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); }; destination Remote_D2 { udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); }; destination Remote_C { udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); }; # # Set local destination for all remote syslog input # destination d_logger { file("/var/log/logger.log"); }; # # Add another local debug file to check filters are working # destination d_debug { file("/var/log/debug.log"); }; ############################################################## # Filters for mimicking Solaris syslogd # # identifying firewall syslogs # # identifying severity levels 0 to 2 # ############################################################## filter f_syslog { facility(mail) or (facility(daemon) and priority(notice)); }; filter f_messages { priority(err) or facility(kern) or (facility(user) and priority(err)) or (facility(daemon) and priority(notice)) or (facility(mail) and priority(crit)); }; filter f_auth { facility (auth); };
# # Set Firewall Filters # filter firewalls { host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); }; filter not_firewalls { not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); };
# # Filter Severity Levels 0, 1 and 2 # filter f_crit { level (crit .. emerg); }; ############################################################## # Now do some logging # ##############################################################
# # Log local syslog data appropriately # log { source(s_local); filter(f_syslog); destination(syslog); }; log { source(s_local); filter(f_messages); destination(messages); }; log { source(s_local); filter(f_auth); destination(loginlog); };
# # Log remotely sourced syslogs of all Severity Levels to remote L system # but exclude syslogs from the firewalls # log { source (src); filter (not_firewalls); destination (Remote_L); destination (d_debug); }; # # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems # including syslogs from the firewalls # #log { # source (src); # filter (f_crit); # destination (Remote_D1); # destination (Remote_D2); #}; # # Log remotely sourced syslogs from the two firewalls to remote C system # #log { #source (src); #filter (firewalls); # destination (Remote_C); #}; # # Log remotely sourced syslogs to a local file # log { source (src); destination (d_logger); };
Hi, Can it happen that syslog-ng becomes CPU bound when you enable the filter? Can you check that using top/vmstat/younameit whether syslog-ng is spinning on a CPU? the host() filter is using regexps, and although your host() pattern doesn't seem complex, it was anonimized in the config, so I don't know if it is more complex in reality. You might want to use a simple substring filter, like so: filter f_firewalls { host("ipaddress" type(string)); }; If you need more than mere substring matching, you might try to use "pcre" (if your syslog-ng is compiled with that), or "glob", which permits shell-like wildcard matches. Otherwise I have no clue why that filter reduces perfomance so much, it shouldn't, but I don't know how much stress you put to it. Hope this helps, Bazsi On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
Initially sent to the wrong address....
Begin forwarded message:
From: Paul Hutton <paul_hutton@bigpond.com>
Subject: Syslog NG simple filtering not working as expected...
Date: 26 September 2013 11:23:15 PM AEST
To: syslog-ng-request@lists.balabit.hu
Hi All,
I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to:
1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out)
In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity.
My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line
filter (not_firewalls);
in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic.
I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site.
I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there...
Thanks in advance.
Paul
@version: 3.2 @include "scl.conf" options { stats_level(3); flush_lines(100); flush_timeout(10000); }; ################################################################ # Ensure syslog-ng logs this host similarly to syslogd # ################################################################ source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # # Set global network source # source src { udp(ip("0.0.0.0") port (514)); };
############################################################### # Local logfiles and remote destinations setup # ############################################################### destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loginlog { file("/var/adm/loginlog"); }; # # Set remote destinations # destination Remote_L { udp ("10.octet.octet.octetL" port (514) ); }; destination Remote_D1 { udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); }; destination Remote_D2 { udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); }; destination Remote_C { udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); }; # # Set local destination for all remote syslog input # destination d_logger { file("/var/log/logger.log"); }; # # Add another local debug file to check filters are working # destination d_debug { file("/var/log/debug.log"); }; ############################################################## # Filters for mimicking Solaris syslogd # # identifying firewall syslogs # # identifying severity levels 0 to 2 # ############################################################## filter f_syslog { facility(mail) or (facility(daemon) and priority(notice)); }; filter f_messages { priority(err) or facility(kern) or (facility(user) and priority(err)) or (facility(daemon) and priority(notice)) or (facility(mail) and priority(crit)); }; filter f_auth { facility (auth); };
# # Set Firewall Filters # filter firewalls { host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); }; filter not_firewalls { not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); };
# # Filter Severity Levels 0, 1 and 2 # filter f_crit { level (crit .. emerg); }; ############################################################## # Now do some logging # ##############################################################
# # Log local syslog data appropriately # log { source(s_local); filter(f_syslog); destination(syslog); }; log { source(s_local); filter(f_messages); destination(messages); }; log { source(s_local); filter(f_auth); destination(loginlog); };
# # Log remotely sourced syslogs of all Severity Levels to remote L system # but exclude syslogs from the firewalls # log { source (src); filter (not_firewalls); destination (Remote_L); destination (d_debug); }; # # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems # including syslogs from the firewalls # #log { # source (src); # filter (f_crit); # destination (Remote_D1); # destination (Remote_D2); #}; # # Log remotely sourced syslogs from the two firewalls to remote C system # #log { #source (src); #filter (firewalls); # destination (Remote_C); #}; # # Log remotely sourced syslogs to a local file # log { source (src); destination (d_logger); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Bazsi, Thanks for your reply. I'm running on an ESX hosted Solaris instance, originally with 1 x 1.8 Ghz CPU and 4 GB RAM. When I enabled diagnostics with syslog-ng-ctl, this pretty much killed the machine, so I'm now running with 2 x 1.8 Ghz CPUs and 8 GB RAM. I only generate a handful of test messages typically 10 to 40, not much of a load. With diagnostics enabled, I have seen entries of 50 or 60 in the vmstat kernel thread wait list, simultaneously with a very idle processor which is a bit of a mystery. However with the additional resources, I don't see any problems. I have just done some further testing and vmstat tells me that the testing had no measurable impact on the system. Syslog-ng-ctl stats never reports any dropped messages to the remote destination, rather the network source "src" messages processed always agrees with the destination "Remote_L" messages processed, as well with the local destination log files. The firewall filter statements contain nothing other than IPV4 addresses. Sorry but I don't understand the distinction that you make between what I am doing with my filter and your suggestion viz:
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
Otherwise with the latest round of tests I suspect that there is something happening at the ESX layer that is messing me up, and which is now being investigated. If this turns out to be the case then I'm sorry for the noise... Thanks and regards Paul On 27/09/2013, at 1:56 AM, Balazs Scheidler wrote:
Hi,
Can it happen that syslog-ng becomes CPU bound when you enable the filter? Can you check that using top/vmstat/younameit whether syslog-ng is spinning on a CPU?
the host() filter is using regexps, and although your host() pattern doesn't seem complex, it was anonimized in the config, so I don't know if it is more complex in reality.
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
If you need more than mere substring matching, you might try to use "pcre" (if your syslog-ng is compiled with that), or "glob", which permits shell-like wildcard matches.
Otherwise I have no clue why that filter reduces perfomance so much, it shouldn't, but I don't know how much stress you put to it.
Hope this helps, Bazsi
On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
Initially sent to the wrong address....
Begin forwarded message:
From: Paul Hutton <paul_hutton@bigpond.com>
Subject: Syslog NG simple filtering not working as expected...
Date: 26 September 2013 11:23:15 PM AEST
To: syslog-ng-request@lists.balabit.hu
Hi All,
I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to:
1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out)
In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity.
My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line
filter (not_firewalls);
in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic.
I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site.
I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there...
Thanks in advance.
Paul
@version: 3.2 @include "scl.conf" options { stats_level(3); flush_lines(100); flush_timeout(10000); }; ################################################################ # Ensure syslog-ng logs this host similarly to syslogd # ################################################################ source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # # Set global network source # source src { udp(ip("0.0.0.0") port (514)); };
############################################################### # Local logfiles and remote destinations setup # ############################################################### destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loginlog { file("/var/adm/loginlog"); }; # # Set remote destinations # destination Remote_L { udp ("10.octet.octet.octetL" port (514) ); }; destination Remote_D1 { udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); }; destination Remote_D2 { udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); }; destination Remote_C { udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); }; # # Set local destination for all remote syslog input # destination d_logger { file("/var/log/logger.log"); }; # # Add another local debug file to check filters are working # destination d_debug { file("/var/log/debug.log"); }; ############################################################## # Filters for mimicking Solaris syslogd # # identifying firewall syslogs # # identifying severity levels 0 to 2 # ############################################################## filter f_syslog { facility(mail) or (facility(daemon) and priority(notice)); }; filter f_messages { priority(err) or facility(kern) or (facility(user) and priority(err)) or (facility(daemon) and priority(notice)) or (facility(mail) and priority(crit)); }; filter f_auth { facility (auth); };
# # Set Firewall Filters # filter firewalls { host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); }; filter not_firewalls { not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); };
# # Filter Severity Levels 0, 1 and 2 # filter f_crit { level (crit .. emerg); }; ############################################################## # Now do some logging # ##############################################################
# # Log local syslog data appropriately # log { source(s_local); filter(f_syslog); destination(syslog); }; log { source(s_local); filter(f_messages); destination(messages); }; log { source(s_local); filter(f_auth); destination(loginlog); };
# # Log remotely sourced syslogs of all Severity Levels to remote L system # but exclude syslogs from the firewalls # log { source (src); filter (not_firewalls); destination (Remote_L); destination (d_debug); }; # # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems # including syslogs from the firewalls # #log { # source (src); # filter (f_crit); # destination (Remote_D1); # destination (Remote_D2); #}; # # Log remotely sourced syslogs from the two firewalls to remote C system # #log { #source (src); #filter (firewalls); # destination (Remote_C); #}; # # Log remotely sourced syslogs to a local file # log { source (src); destination (d_logger); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi again, Further to this it would appear that the network destination logger doesn't deal well with the syslog format that is being forwarded to it from the syslog-ng host. In particular, the syslog-ng host receives syslogs from the network source which have the fields: "local date" ,"local time", "hostname", "UTC date" ,"UTC time", "message string" The syslog-ng host then forwards to the network destination the above data, but with it prepended by a second entry of "local date","local time" and "source ip address", rather than hostname. I have been using spoof_source (yes) in the destination definition. Will this create the duplicate date and time entry as well as the source ip address? Thanks Paul On 27/09/2013, at 1:26 PM, Paul Hutton wrote:
Hi Bazsi,
Thanks for your reply. I'm running on an ESX hosted Solaris instance, originally with 1 x 1.8 Ghz CPU and 4 GB RAM. When I enabled diagnostics with syslog-ng-ctl, this pretty much killed the machine, so I'm now running with 2 x 1.8 Ghz CPUs and 8 GB RAM. I only generate a handful of test messages typically 10 to 40, not much of a load.
With diagnostics enabled, I have seen entries of 50 or 60 in the vmstat kernel thread wait list, simultaneously with a very idle processor which is a bit of a mystery. However with the additional resources, I don't see any problems. I have just done some further testing and vmstat tells me that the testing had no measurable impact on the system. Syslog-ng-ctl stats never reports any dropped messages to the remote destination, rather the network source "src" messages processed always agrees with the destination "Remote_L" messages processed, as well with the local destination log files.
The firewall filter statements contain nothing other than IPV4 addresses.
Sorry but I don't understand the distinction that you make between what I am doing with my filter and your suggestion viz:
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
Otherwise with the latest round of tests I suspect that there is something happening at the ESX layer that is messing me up, and which is now being investigated. If this turns out to be the case then I'm sorry for the noise...
Thanks and regards
Paul
On 27/09/2013, at 1:56 AM, Balazs Scheidler wrote:
Hi,
Can it happen that syslog-ng becomes CPU bound when you enable the filter? Can you check that using top/vmstat/younameit whether syslog-ng is spinning on a CPU?
the host() filter is using regexps, and although your host() pattern doesn't seem complex, it was anonimized in the config, so I don't know if it is more complex in reality.
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
If you need more than mere substring matching, you might try to use "pcre" (if your syslog-ng is compiled with that), or "glob", which permits shell-like wildcard matches.
Otherwise I have no clue why that filter reduces perfomance so much, it shouldn't, but I don't know how much stress you put to it.
Hope this helps, Bazsi
On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
Initially sent to the wrong address....
Begin forwarded message:
From: Paul Hutton <paul_hutton@bigpond.com>
Subject: Syslog NG simple filtering not working as expected...
Date: 26 September 2013 11:23:15 PM AEST
To: syslog-ng-request@lists.balabit.hu
Hi All,
I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to:
1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out)
In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity.
My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line
filter (not_firewalls);
in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic.
I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site.
I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there...
Thanks in advance.
Paul
@version: 3.2 @include "scl.conf" options { stats_level(3); flush_lines(100); flush_timeout(10000); }; ################################################################ # Ensure syslog-ng logs this host similarly to syslogd # ################################################################ source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # # Set global network source # source src { udp(ip("0.0.0.0") port (514)); };
############################################################### # Local logfiles and remote destinations setup # ############################################################### destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loginlog { file("/var/adm/loginlog"); }; # # Set remote destinations # destination Remote_L { udp ("10.octet.octet.octetL" port (514) ); }; destination Remote_D1 { udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); }; destination Remote_D2 { udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); }; destination Remote_C { udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); }; # # Set local destination for all remote syslog input # destination d_logger { file("/var/log/logger.log"); }; # # Add another local debug file to check filters are working # destination d_debug { file("/var/log/debug.log"); }; ############################################################## # Filters for mimicking Solaris syslogd # # identifying firewall syslogs # # identifying severity levels 0 to 2 # ############################################################## filter f_syslog { facility(mail) or (facility(daemon) and priority(notice)); }; filter f_messages { priority(err) or facility(kern) or (facility(user) and priority(err)) or (facility(daemon) and priority(notice)) or (facility(mail) and priority(crit)); }; filter f_auth { facility (auth); };
# # Set Firewall Filters # filter firewalls { host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); }; filter not_firewalls { not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); };
# # Filter Severity Levels 0, 1 and 2 # filter f_crit { level (crit .. emerg); }; ############################################################## # Now do some logging # ##############################################################
# # Log local syslog data appropriately # log { source(s_local); filter(f_syslog); destination(syslog); }; log { source(s_local); filter(f_messages); destination(messages); }; log { source(s_local); filter(f_auth); destination(loginlog); };
# # Log remotely sourced syslogs of all Severity Levels to remote L system # but exclude syslogs from the firewalls # log { source (src); filter (not_firewalls); destination (Remote_L); destination (d_debug); }; # # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems # including syslogs from the firewalls # #log { # source (src); # filter (f_crit); # destination (Remote_D1); # destination (Remote_D2); #}; # # Log remotely sourced syslogs from the two firewalls to remote C system # #log { #source (src); #filter (firewalls); # destination (Remote_C); #}; # # Log remotely sourced syslogs to a local file # log { source (src); destination (d_logger); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You should check the incoming traffic syslog-ng gets to see whether it is in a known and supported format or something which only resembles to syslog traffic so it could happen that syslog-ng doesn't parse it properly. What Bazsi referred to is that processing an IP address (containing dots) as a regexp is more expensive than using it for a plain string comparision. It's a micro-optimisation though. spoofing doesn't alter the message part, it changes the source IP address of the outgoing packet this is why it works only for UDP. On Wed, Oct 2, 2013 at 3:24 PM, Paul Hutton <paul_hutton@bigpond.com> wrote:
Hi again,
Further to this it would appear that the network destination logger doesn't deal well with the syslog format that is being forwarded to it from the syslog-ng host. In particular, the syslog-ng host receives syslogs from the network source which have the fields:
"local date" ,"local time", "hostname", "UTC date" ,"UTC time", "message string"
The syslog-ng host then forwards to the network destination the above data, but with it prepended by a second entry of "local date","local time" and "source ip address", rather than hostname.
I have been using spoof_source (yes) in the destination definition. Will this create the duplicate date and time entry as well as the source ip address?
Thanks
Paul
On 27/09/2013, at 1:26 PM, Paul Hutton wrote:
Hi Bazsi,
Thanks for your reply. I'm running on an ESX hosted Solaris instance, originally with 1 x 1.8 Ghz CPU and 4 GB RAM. When I enabled diagnostics with syslog-ng-ctl, this pretty much killed the machine, so I'm now running with 2 x 1.8 Ghz CPUs and 8 GB RAM. I only generate a handful of test messages typically 10 to 40, not much of a load.
With diagnostics enabled, I have seen entries of 50 or 60 in the vmstat kernel thread wait list, simultaneously with a very idle processor which is a bit of a mystery. However with the additional resources, I don't see any problems. I have just done some further testing and vmstat tells me that the testing had no measurable impact on the system. Syslog-ng-ctl stats never reports any dropped messages to the remote destination, rather the network source "src" messages processed always agrees with the destination "Remote_L" messages processed, as well with the local destination log files.
The firewall filter statements contain nothing other than IPV4 addresses.
Sorry but I don't understand the distinction that you make between what I am doing with my filter and your suggestion viz:
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
Otherwise with the latest round of tests I suspect that there is something happening at the ESX layer that is messing me up, and which is now being investigated. If this turns out to be the case then I'm sorry for the noise...
Thanks and regards
Paul
On 27/09/2013, at 1:56 AM, Balazs Scheidler wrote:
Hi,
Can it happen that syslog-ng becomes CPU bound when you enable the filter? Can you check that using top/vmstat/younameit whether syslog-ng is spinning on a CPU?
the host() filter is using regexps, and although your host() pattern doesn't seem complex, it was anonimized in the config, so I don't know if it is more complex in reality.
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
If you need more than mere substring matching, you might try to use "pcre" (if your syslog-ng is compiled with that), or "glob", which permits shell-like wildcard matches.
Otherwise I have no clue why that filter reduces perfomance so much, it shouldn't, but I don't know how much stress you put to it.
Hope this helps, Bazsi
On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
Initially sent to the wrong address....
Begin forwarded message:
From: Paul Hutton <paul_hutton@bigpond.com>
Subject: Syslog NG simple filtering not working as expected...
Date: 26 September 2013 11:23:15 PM AEST
To: syslog-ng-request@lists.balabit.hu
Hi All,
I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to:
1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out)
In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity.
My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line
filter (not_firewalls);
in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic.
I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site.
I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there...
Thanks in advance.
Paul
@version: 3.2 @include "scl.conf" options { stats_level(3); flush_lines(100); flush_timeout(10000); }; ################################################################ # Ensure syslog-ng logs this host similarly to syslogd # ################################################################ source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # # Set global network source # source src { udp(ip("0.0.0.0") port (514)); };
############################################################### # Local logfiles and remote destinations setup # ############################################################### destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loginlog { file("/var/adm/loginlog"); }; # # Set remote destinations # destination Remote_L { udp ("10.octet.octet.octetL" port (514) ); }; destination Remote_D1 { udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); }; destination Remote_D2 { udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); }; destination Remote_C { udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); }; # # Set local destination for all remote syslog input # destination d_logger { file("/var/log/logger.log"); }; # # Add another local debug file to check filters are working # destination d_debug { file("/var/log/debug.log"); }; ############################################################## # Filters for mimicking Solaris syslogd # # identifying firewall syslogs # # identifying severity levels 0 to 2 # ############################################################## filter f_syslog { facility(mail) or (facility(daemon) and priority(notice)); }; filter f_messages { priority(err) or facility(kern) or (facility(user) and priority(err)) or (facility(daemon) and priority(notice)) or (facility(mail) and priority(crit)); }; filter f_auth { facility (auth); };
# # Set Firewall Filters # filter firewalls { host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); }; filter not_firewalls { not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); };
# # Filter Severity Levels 0, 1 and 2 # filter f_crit { level (crit .. emerg); }; ############################################################## # Now do some logging # ##############################################################
# # Log local syslog data appropriately # log { source(s_local); filter(f_syslog); destination(syslog); }; log { source(s_local); filter(f_messages); destination(messages); }; log { source(s_local); filter(f_auth); destination(loginlog); };
# # Log remotely sourced syslogs of all Severity Levels to remote L system # but exclude syslogs from the firewalls # log { source (src); filter (not_firewalls); destination (Remote_L); destination (d_debug); }; # # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems # including syslogs from the firewalls # #log { # source (src); # filter (f_crit); # destination (Remote_D1); # destination (Remote_D2); #}; # # Log remotely sourced syslogs from the two firewalls to remote C system # #log { #source (src); #filter (firewalls); # destination (Remote_C); #}; # # Log remotely sourced syslogs to a local file # log { source (src); destination (d_logger); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Not sure if this applies to your specific needs but we do a couple things that sound similar. First, we find that anything that causes DNS resolution really slows things down. So we put some common options together like this: options { # check if the hostname contains valid characters check_hostname(yes); # do not resolve names for speed use_dns(no); dns_cache(no); # keep hostnames to enable related macros keep_hostname(yes); # do not track / forward syslog forwarder chain chain_hostnames(no); # options related to file and directories dir_owner("root"); dir_group("log_ro"); owner("root"); group("log_ro"); perm(0640); dir_perm(0750); create_dirs(yes); }; Second - we filter based on netmask since we believe it is easier than any text matching (should be able to be done with a bitmask on the IP header and not even need to parse the message at all) #!# filter for specific servers - redacted with fake addressses filter f_filter1 { netmask( "192.168.1.1/32" ) or netmask( "192.168.2.2/32" ) or netmask( "10.0.1.1/32" ) or netmask( "10.0.2.2/32" ); }; If this helps - you're welcome. If not - my apologies :-) Jim ---- Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
You should check the incoming traffic syslog-ng gets to see whether it is in a known and supported format or something which only resembles to syslog traffic so it could happen that syslog-ng doesn't parse it properly.
What Bazsi referred to is that processing an IP address (containing dots) as a regexp is more expensive than using it for a plain string comparision. It's a micro-optimisation though.
spoofing doesn't alter the message part, it changes the source IP address of the outgoing packet this is why it works only for UDP.
On Wed, Oct 2, 2013 at 3:24 PM, Paul Hutton <paul_hutton@bigpond.com> wrote:
Hi again,
Further to this it would appear that the network destination logger doesn't deal well with the syslog format that is being forwarded to it from the syslog-ng host. In particular, the syslog-ng host receives syslogs from the network source which have the fields:
"local date" ,"local time", "hostname", "UTC date" ,"UTC time", "message string"
The syslog-ng host then forwards to the network destination the above data, but with it prepended by a second entry of "local date","local time" and "source ip address", rather than hostname.
I have been using spoof_source (yes) in the destination definition. Will this create the duplicate date and time entry as well as the source ip address?
Thanks
Paul
On 27/09/2013, at 1:26 PM, Paul Hutton wrote:
Hi Bazsi,
Thanks for your reply. I'm running on an ESX hosted Solaris instance, originally with 1 x 1.8 Ghz CPU and 4 GB RAM. When I enabled diagnostics with syslog-ng-ctl, this pretty much killed the machine, so I'm now running with 2 x 1.8 Ghz CPUs and 8 GB RAM. I only generate a handful of test messages typically 10 to 40, not much of a load.
With diagnostics enabled, I have seen entries of 50 or 60 in the vmstat kernel thread wait list, simultaneously with a very idle processor which is a bit of a mystery. However with the additional resources, I don't see any problems. I have just done some further testing and vmstat tells me that the testing had no measurable impact on the system. Syslog-ng-ctl stats never reports any dropped messages to the remote destination, rather the network source "src" messages processed always agrees with the destination "Remote_L" messages processed, as well with the local destination log files.
The firewall filter statements contain nothing other than IPV4 addresses.
Sorry but I don't understand the distinction that you make between what I am doing with my filter and your suggestion viz:
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
Otherwise with the latest round of tests I suspect that there is something happening at the ESX layer that is messing me up, and which is now being investigated. If this turns out to be the case then I'm sorry for the noise...
Thanks and regards
Paul
On 27/09/2013, at 1:56 AM, Balazs Scheidler wrote:
Hi,
Can it happen that syslog-ng becomes CPU bound when you enable the filter? Can you check that using top/vmstat/younameit whether syslog-ng is spinning on a CPU?
the host() filter is using regexps, and although your host() pattern doesn't seem complex, it was anonimized in the config, so I don't know if it is more complex in reality.
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
If you need more than mere substring matching, you might try to use "pcre" (if your syslog-ng is compiled with that), or "glob", which permits shell-like wildcard matches.
Otherwise I have no clue why that filter reduces perfomance so much, it shouldn't, but I don't know how much stress you put to it.
Hope this helps, Bazsi
On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
Initially sent to the wrong address....
Begin forwarded message:
From: Paul Hutton <paul_hutton@bigpond.com>
Subject: Syslog NG simple filtering not working as expected...
Date: 26 September 2013 11:23:15 PM AEST
To: syslog-ng-request@lists.balabit.hu
Hi All,
I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to:
1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out)
In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity.
My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line
filter (not_firewalls);
in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic.
I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site.
I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there...
Thanks in advance.
Paul
@version: 3.2 @include "scl.conf" options { stats_level(3); flush_lines(100); flush_timeout(10000); }; ################################################################ # Ensure syslog-ng logs this host similarly to syslogd # ################################################################ source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # # Set global network source # source src { udp(ip("0.0.0.0") port (514)); };
############################################################### # Local logfiles and remote destinations setup # ############################################################### destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loginlog { file("/var/adm/loginlog"); }; # # Set remote destinations # destination Remote_L { udp ("10.octet.octet.octetL" port (514) ); }; destination Remote_D1 { udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); }; destination Remote_D2 { udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); }; destination Remote_C { udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); }; # # Set local destination for all remote syslog input # destination d_logger { file("/var/log/logger.log"); }; # # Add another local debug file to check filters are working # destination d_debug { file("/var/log/debug.log"); }; ############################################################## # Filters for mimicking Solaris syslogd # # identifying firewall syslogs # # identifying severity levels 0 to 2 # ############################################################## filter f_syslog { facility(mail) or (facility(daemon) and priority(notice)); }; filter f_messages { priority(err) or facility(kern) or (facility(user) and priority(err)) or (facility(daemon) and priority(notice)) or (facility(mail) and priority(crit)); }; filter f_auth { facility (auth); };
# # Set Firewall Filters # filter firewalls { host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); }; filter not_firewalls { not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); };
# # Filter Severity Levels 0, 1 and 2 # filter f_crit { level (crit .. emerg); }; ############################################################## # Now do some logging # ##############################################################
# # Log local syslog data appropriately # log { source(s_local); filter(f_syslog); destination(syslog); }; log { source(s_local); filter(f_messages); destination(messages); }; log { source(s_local); filter(f_auth); destination(loginlog); };
# # Log remotely sourced syslogs of all Severity Levels to remote L system # but exclude syslogs from the firewalls # log { source (src); filter (not_firewalls); destination (Remote_L); destination (d_debug); }; # # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems # including syslogs from the firewalls # #log { # source (src); # filter (f_crit); # destination (Remote_D1); # destination (Remote_D2); #}; # # Log remotely sourced syslogs from the two firewalls to remote C system # #log { #source (src); #filter (firewalls); # destination (Remote_C); #}; # # Log remotely sourced syslogs to a local file # log { source (src); destination (d_logger); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balazs Scheidler
-
jrhendri＠roadrunner.com
-
Paul Hutton
-
Sandor Geller