[syslog-ng] Fwd: Syslog NG simple filtering not working as expected...
Paul Hutton
paul_hutton at bigpond.com
Fri Sep 27 05:26:21 CEST 2013
Hi Bazsi,
Thanks for your reply. I'm running on an ESX hosted Solaris instance, originally with 1 x 1.8 Ghz CPU and 4 GB RAM. When I enabled diagnostics with syslog-ng-ctl, this pretty much killed the machine, so I'm now running with 2 x 1.8 Ghz CPUs and 8 GB RAM. I only generate a handful of test messages typically 10 to 40, not much of a load.
With diagnostics enabled, I have seen entries of 50 or 60 in the vmstat kernel thread wait list, simultaneously with a very idle processor which is a bit of a mystery. However with the additional resources, I don't see any problems. I have just done some further testing and vmstat tells me that the testing had no measurable impact on the system. Syslog-ng-ctl stats never reports any dropped messages to the remote destination, rather the network source "src" messages processed always agrees with the destination "Remote_L" messages processed, as well with the local destination log files.
The firewall filter statements contain nothing other than IPV4 addresses.
Sorry but I don't understand the distinction that you make between what I am doing with my filter and your suggestion viz:
> You might want to use a simple substring filter, like so:
>
> filter f_firewalls { host("ipaddress" type(string)); };
Otherwise with the latest round of tests I suspect that there is something happening at the ESX layer that is messing me up, and which is now being investigated. If this turns out to be the case then I'm sorry for the noise...
Thanks and regards
Paul
On 27/09/2013, at 1:56 AM, Balazs Scheidler wrote:
> Hi,
>
> Can it happen that syslog-ng becomes CPU bound when you enable the
> filter? Can you check that using top/vmstat/younameit whether syslog-ng
> is spinning on a CPU?
>
> the host() filter is using regexps, and although your host() pattern
> doesn't seem complex, it was anonimized in the config, so I don't know
> if it is more complex in reality.
>
> You might want to use a simple substring filter, like so:
>
> filter f_firewalls { host("ipaddress" type(string)); };
>
> If you need more than mere substring matching, you might try to use
> "pcre" (if your syslog-ng is compiled with that), or "glob", which
> permits shell-like wildcard matches.
>
> Otherwise I have no clue why that filter reduces perfomance so much, it
> shouldn't, but I don't know how much stress you put to it.
>
> Hope this helps,
> Bazsi
>
>
> On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
>> Initially sent to the wrong address....
>>
>> Begin forwarded message:
>>
>>> From: Paul Hutton <paul_hutton at bigpond.com>
>>>
>>> Subject: Syslog NG simple filtering not working as expected...
>>>
>>> Date: 26 September 2013 11:23:15 PM AEST
>>>
>>> To: syslog-ng-request at lists.balabit.hu
>>>
>>>
>>> Hi All,
>>>
>>>
>>> I am trying to forward network sourced syslogs to a number of
>>> destinations depending on a few simple filters, but am immediately
>>> coming unstuck. My constraint is that all incoming and outgoing
>>> network syslog traffic is udp. My intention in the configuration
>>> code below (apart from logging the localhost) is to:
>>>
>>>
>>> 1. Forward all incoming network sourced syslogs except those
>>> generated by two firewalls to destination Remote_L
>>> 2. Forward incoming network syslogs that are generated by the
>>> firewalls to destination Remote_C (currently commented out)
>>> 3. Forward all incoming network syslogs of severity 0, 1 and 2 to
>>> destinations Remote_D1 and Remote_D2 (currently commented out)
>>>
>>>
>>> In addition all incoming network sourced syslogs are logged locally
>>> to destination d_logger and I've added another local file d_debug
>>> out of curiosity.
>>>
>>>
>>> My problem is that with just the filter enabled at point 1 above and
>>> in the code below, the behaviour of syslog-ng is very erratic,
>>> beyond any reasonable anticipation of the unreliability of udp as
>>> the transport. If I comment out the filter line
>>>
>>>
>>> filter (not_firewalls);
>>>
>>>
>>> in the code below, then test traffic (none of which is generated by
>>> the firewall sources) is received and forwarded correctly as well
>>> as logged locally. If I uncomment the filter, then I have seen test
>>> traffic forwarded correctly on one occasion, but it is mostly the
>>> case that some smaller subset (including none) of the test traffic
>>> is forwarded. In this case, the test traffic is also correctly
>>> written to the local destination d_debug. That is in the same rule
>>> that fails to forward the test traffic.
>>>
>>>
>>> I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the
>>> Sunfreeware site.
>>>
>>>
>>> I have also unashamedly plagiarised configuration code from a couple
>>> of posted sources, so if you recognise your work, thanks for putting
>>> it out there...
>>>
>>>
>>>
>>>
>>> Thanks in advance.
>>>
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>>
>>>
>>> @version: 3.2
>>> @include "scl.conf"
>>> options {
>>> stats_level(3);
>>> flush_lines(100);
>>> flush_timeout(10000);
>>> };
>>> ################################################################
>>> # Ensure syslog-ng logs this host similarly to
>>> syslogd #
>>> ################################################################
>>> source s_local {
>>> sun-streams("/dev/log" door("/etc/.syslog_door"));
>>> internal();
>>> };
>>> #
>>> # Set global network source
>>> #
>>> source src {
>>> udp(ip("0.0.0.0") port (514));
>>> };
>>>
>>> ###############################################################
>>> # Local logfiles and remote destinations
>>> setup #
>>> ###############################################################
>>> destination syslog {
>>> file("/var/log/syslog");
>>> };
>>> destination messages {
>>> file("/var/adm/messages");
>>> };
>>> destination loginlog {
>>> file("/var/adm/loginlog");
>>> };
>>> #
>>> # Set remote destinations
>>> #
>>> destination Remote_L {
>>> udp ("10.octet.octet.octetL" port (514) );
>>> };
>>> destination Remote_D1 {
>>> udp ("10.octet.octet.octetD1" port (514) spoof_source
>>> (yes));
>>> };
>>> destination Remote_D2 {
>>> udp ("10.octet.octet.octetD2" port (514) spoof_source
>>> (yes));
>>> };
>>> destination Remote_C {
>>> udp ("10.octet.octet.octetC" port (514) spoof_source (yes));
>>> };
>>> #
>>> # Set local destination for all remote syslog input
>>> #
>>> destination d_logger {
>>> file("/var/log/logger.log");
>>> };
>>> #
>>> # Add another local debug file to check filters are working
>>> #
>>> destination d_debug {
>>> file("/var/log/debug.log");
>>> };
>>> ##############################################################
>>> # Filters for mimicking Solaris
>>> syslogd #
>>> # identifying firewall
>>> syslogs #
>>> # identifying severity levels 0 to
>>> 2 #
>>> ##############################################################
>>> filter f_syslog {
>>> facility(mail) or (facility(daemon) and priority(notice));
>>> };
>>> filter f_messages {
>>> priority(err)
>>> or facility(kern)
>>> or (facility(user) and priority(err))
>>> or (facility(daemon) and priority(notice))
>>> or (facility(mail) and priority(crit));
>>> };
>>> filter f_auth {
>>> facility (auth);
>>> };
>>>
>>> #
>>> # Set Firewall Filters
>>> #
>>> filter firewalls {
>>> host("10.octet.octet.octetfw1") or
>>> host("10.octet.octet.octetfw2");
>>> };
>>> filter not_firewalls {
>>> not (host("10.octet.octet.octetfw1") or
>>> host("10.octet.octet.octetfw2"));
>>> };
>>>
>>> #
>>> # Filter Severity Levels 0, 1 and 2
>>> #
>>> filter f_crit {
>>> level (crit .. emerg);
>>> };
>>> ##############################################################
>>> # Now do some logging #
>>> ##############################################################
>>>
>>> #
>>> # Log local syslog data appropriately
>>> #
>>> log {
>>> source(s_local);
>>> filter(f_syslog);
>>> destination(syslog);
>>> };
>>> log {
>>> source(s_local);
>>> filter(f_messages);
>>> destination(messages);
>>> };
>>> log {
>>> source(s_local);
>>> filter(f_auth);
>>> destination(loginlog);
>>> };
>>>
>>> #
>>> # Log remotely sourced syslogs of all Severity Levels to
>>> remote L system
>>> # but exclude syslogs from the firewalls
>>> #
>>> log {
>>> source (src);
>>> filter (not_firewalls);
>>> destination (Remote_L);
>>> destination (d_debug);
>>> };
>>> #
>>> # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D
>>> systems
>>> # including syslogs from the firewalls
>>> #
>>> #log {
>>> # source (src);
>>> # filter (f_crit);
>>> # destination (Remote_D1);
>>> # destination (Remote_D2);
>>> #};
>>> #
>>> # Log remotely sourced syslogs from the two firewalls to remote
>>> C system
>>> #
>>> #log {
>>> #source (src);
>>> #filter (firewalls);
>>> # destination (Remote_C);
>>> #};
>>> #
>>> # Log remotely sourced syslogs to a local file
>>> #
>>> log {
>>> source (src);
>>> destination (d_logger);
>>> };
>>>
>>>
>>>
>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
More information about the syslog-ng
mailing list