[syslog-ng] Fwd: Syslog NG simple filtering not working as expected...
Balazs Scheidler
bazsi at balabit.hu
Thu Sep 26 17:56:48 CEST 2013
Hi,
Can it happen that syslog-ng becomes CPU bound when you enable the
filter? Can you check that using top/vmstat/younameit whether syslog-ng
is spinning on a CPU?
the host() filter is using regexps, and although your host() pattern
doesn't seem complex, it was anonimized in the config, so I don't know
if it is more complex in reality.
You might want to use a simple substring filter, like so:
filter f_firewalls { host("ipaddress" type(string)); };
If you need more than mere substring matching, you might try to use
"pcre" (if your syslog-ng is compiled with that), or "glob", which
permits shell-like wildcard matches.
Otherwise I have no clue why that filter reduces perfomance so much, it
shouldn't, but I don't know how much stress you put to it.
Hope this helps,
Bazsi
On Thu, 2013-09-26 at 23:27 +1000, Paul Hutton wrote:
> Initially sent to the wrong address....
>
> Begin forwarded message:
>
> > From: Paul Hutton <paul_hutton at bigpond.com>
> >
> > Subject: Syslog NG simple filtering not working as expected...
> >
> > Date: 26 September 2013 11:23:15 PM AEST
> >
> > To: syslog-ng-request at lists.balabit.hu
> >
> >
> > Hi All,
> >
> >
> > I am trying to forward network sourced syslogs to a number of
> > destinations depending on a few simple filters, but am immediately
> > coming unstuck. My constraint is that all incoming and outgoing
> > network syslog traffic is udp. My intention in the configuration
> > code below (apart from logging the localhost) is to:
> >
> >
> > 1. Forward all incoming network sourced syslogs except those
> > generated by two firewalls to destination Remote_L
> > 2. Forward incoming network syslogs that are generated by the
> > firewalls to destination Remote_C (currently commented out)
> > 3. Forward all incoming network syslogs of severity 0, 1 and 2 to
> > destinations Remote_D1 and Remote_D2 (currently commented out)
> >
> >
> > In addition all incoming network sourced syslogs are logged locally
> > to destination d_logger and I've added another local file d_debug
> > out of curiosity.
> >
> >
> > My problem is that with just the filter enabled at point 1 above and
> > in the code below, the behaviour of syslog-ng is very erratic,
> > beyond any reasonable anticipation of the unreliability of udp as
> > the transport. If I comment out the filter line
> >
> >
> > filter (not_firewalls);
> >
> >
> > in the code below, then test traffic (none of which is generated by
> > the firewall sources) is received and forwarded correctly as well
> > as logged locally. If I uncomment the filter, then I have seen test
> > traffic forwarded correctly on one occasion, but it is mostly the
> > case that some smaller subset (including none) of the test traffic
> > is forwarded. In this case, the test traffic is also correctly
> > written to the local destination d_debug. That is in the same rule
> > that fails to forward the test traffic.
> >
> >
> > I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the
> > Sunfreeware site.
> >
> >
> > I have also unashamedly plagiarised configuration code from a couple
> > of posted sources, so if you recognise your work, thanks for putting
> > it out there...
> >
> >
> >
> >
> > Thanks in advance.
> >
> >
> > Paul
> >
> >
> >
> >
> >
> >
> > @version: 3.2
> > @include "scl.conf"
> > options {
> > stats_level(3);
> > flush_lines(100);
> > flush_timeout(10000);
> > };
> > ################################################################
> > # Ensure syslog-ng logs this host similarly to
> > syslogd #
> > ################################################################
> > source s_local {
> > sun-streams("/dev/log" door("/etc/.syslog_door"));
> > internal();
> > };
> > #
> > # Set global network source
> > #
> > source src {
> > udp(ip("0.0.0.0") port (514));
> > };
> >
> > ###############################################################
> > # Local logfiles and remote destinations
> > setup #
> > ###############################################################
> > destination syslog {
> > file("/var/log/syslog");
> > };
> > destination messages {
> > file("/var/adm/messages");
> > };
> > destination loginlog {
> > file("/var/adm/loginlog");
> > };
> > #
> > # Set remote destinations
> > #
> > destination Remote_L {
> > udp ("10.octet.octet.octetL" port (514) );
> > };
> > destination Remote_D1 {
> > udp ("10.octet.octet.octetD1" port (514) spoof_source
> > (yes));
> > };
> > destination Remote_D2 {
> > udp ("10.octet.octet.octetD2" port (514) spoof_source
> > (yes));
> > };
> > destination Remote_C {
> > udp ("10.octet.octet.octetC" port (514) spoof_source (yes));
> > };
> > #
> > # Set local destination for all remote syslog input
> > #
> > destination d_logger {
> > file("/var/log/logger.log");
> > };
> > #
> > # Add another local debug file to check filters are working
> > #
> > destination d_debug {
> > file("/var/log/debug.log");
> > };
> > ##############################################################
> > # Filters for mimicking Solaris
> > syslogd #
> > # identifying firewall
> > syslogs #
> > # identifying severity levels 0 to
> > 2 #
> > ##############################################################
> > filter f_syslog {
> > facility(mail) or (facility(daemon) and priority(notice));
> > };
> > filter f_messages {
> > priority(err)
> > or facility(kern)
> > or (facility(user) and priority(err))
> > or (facility(daemon) and priority(notice))
> > or (facility(mail) and priority(crit));
> > };
> > filter f_auth {
> > facility (auth);
> > };
> >
> > #
> > # Set Firewall Filters
> > #
> > filter firewalls {
> > host("10.octet.octet.octetfw1") or
> > host("10.octet.octet.octetfw2");
> > };
> > filter not_firewalls {
> > not (host("10.octet.octet.octetfw1") or
> > host("10.octet.octet.octetfw2"));
> > };
> >
> > #
> > # Filter Severity Levels 0, 1 and 2
> > #
> > filter f_crit {
> > level (crit .. emerg);
> > };
> > ##############################################################
> > # Now do some logging #
> > ##############################################################
> >
> > #
> > # Log local syslog data appropriately
> > #
> > log {
> > source(s_local);
> > filter(f_syslog);
> > destination(syslog);
> > };
> > log {
> > source(s_local);
> > filter(f_messages);
> > destination(messages);
> > };
> > log {
> > source(s_local);
> > filter(f_auth);
> > destination(loginlog);
> > };
> >
> > #
> > # Log remotely sourced syslogs of all Severity Levels to
> > remote L system
> > # but exclude syslogs from the firewalls
> > #
> > log {
> > source (src);
> > filter (not_firewalls);
> > destination (Remote_L);
> > destination (d_debug);
> > };
> > #
> > # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D
> > systems
> > # including syslogs from the firewalls
> > #
> > #log {
> > # source (src);
> > # filter (f_crit);
> > # destination (Remote_D1);
> > # destination (Remote_D2);
> > #};
> > #
> > # Log remotely sourced syslogs from the two firewalls to remote
> > C system
> > #
> > #log {
> > #source (src);
> > #filter (firewalls);
> > # destination (Remote_C);
> > #};
> > #
> > # Log remotely sourced syslogs to a local file
> > #
> > log {
> > source (src);
> > destination (d_logger);
> > };
> >
> >
> >
> >
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
More information about the syslog-ng
mailing list