[syslog-ng] dropped udp packets and help with config

[Sandor Geller]

Hello,

On Fri, May 13, 2011 at 7:43 PM, Zeek Anow <zeekstern at gmail.com> wrote:
>
> Something really wrong with syslog-ng or my config. I'm dropping way too
> many packets.
> I will admit that my configuration is probably really a large part of the
> problem
> and would appreciate it if someone could take a look at it and offer some
> suggestions.
>  There is another thread going about a similar problem on a similar
> platform.
>
>  We recently upgraded to Solaris 10 from Solaris 9 and I don't recall us
> dropping that
> many packets before. And we also upgraded from a very older Sylog-ng version
> to 3.1.2.
> I am basing the dropped packets on the udp stats, not syslog-ng stats.
> Syslog-ng stats has NO dropped packets.

This implies that syslog-ng couldn't read the messages from the UDP
socket so the kernel dropped incoming logs to the floor.

> UDP     udpInDatagrams      -4599313    udpInErrors         -     0
>         udpOutDatagrams     -  3421     udpOutErrors        -     0
>         tcpInErrs           -     0     udpNoPorts          -2587612
>         udpInCksumErrs      -     0     udpInOverflows      -95806254
>

...

> Below is a very small sampling of our syslog-ng.conf. We are filtering on
> about
> 1400 devices most of which are Firewalls and routers. The IPs in the
> following
> sample have been made up.

1400 *distinct* log sources like TCP streams? A single-threaded
syslog-ng instance might not scale for such amount of log sources. An
option would be running multiple syslog-ng instances listening on
different ports and configure clients to use different ports sharing
the load between the syslog-ng instances.

> One of my questions is "Does the number of devices we are filtering on make
> a difference? (1400)"
> We have several sites and use just one version of the syslogng.conf file. It
> is
> a lot easier to maintain one copy:))
>
> Also notice the format:  ("^10\.123\.10\.133$") for the filters. All 1400
> are in that format.
>  I was hoping this would help a little but don't really know for sure:))

Could you make your filters hierarchical? Using the netmask() filter
and nested log statements you could reduce the number of filter
evaluations. For example

filter f_tenten {
  netmask("10.10/16";)
};

filter tentenone {
  netmask("10.10.1/24");
};

filter tententwo {
  netmask("10.10.2/24");
};

filter f_teneleven {
  netmask("10.11/16";)
};

...

and later

log {
  # first big network
  source(...);
  filter(f_tenten);
  log {
    filter(f_tentenone);
    destination(...);
  };
  log {
    filter(tententwo);
    destination(...);
  };
  flags(final);
};

log {
  # second big network
  source(...);
  filter(f_teneleven);
  ...
};

and so on, organizing filters into tree / forest hierarchy instead of
evaluating all filters one by one for all messages. This way logs
coming from 10.10/16 will get processed in the first log block. Logs
originating from other networks will trigger the evaluation of the
f_tenten filter and as it will give a false result none of the
embedded log statements (including their filters) will process the
log. So the logs will get processed in the next log section (with the
second big network comment).

Using flags(final) is very important here, it tells syslog-ng to do
not process more log{} sections for the given log message, otherwise
all further log sections will process the message. If you want to log
a message to multiple places then just add the additional destinations
to the proper log{} block.

Regards,

Sandor