[syslog-ng] dropped udp packets and help with config

[Zeek Anow]

Thanks for the reply Sandor. Much appreciated!1

I moved your comments up here to make it a little easier.

Syslog-ng not being able to read the messages from UDP socket happens
very fast. Probably within a minute or less of turning syslog-ng on.
I have the only 2 Solaris parameters that I can set, (recv_high_water and
udp_max_buf) set to the highest limit possible. I was thinking that
something on the syslog-ng side might help. log_fetch_limit, log_iw_size
log_fifo, flush_lines etc might help, but I don't have a clue of
where to start or what to set them to. I could spend months pulling
numbers out of the air and plugging them in to see what happens:))

With regards to your "1400 *distinct* log sources like TCP streams?".
Up until a few weeks ago, we did have about 8 devices that were
distinct log sources like TCP streams. This accounted for about
60% of our traffic. What we found was that a few devices
were about 6 days behind in the log files. We changed their
method of logging so it does not use syslog-ng anymore.

I really like your idea of making the filters hierarchical and
appreciate your example. That makes sense to me. At one time,
we did do something similar to that but used a lot of regex to
accomplish it. That is when we changed and went the
("^10\.123\.10\.133$") route for each IP address, thinking it
would be more efficient. I will check into this asap!!

Regards,
Zeek

On Fri, May 13, 2011 at 2:44 PM, Sandor Geller <
Sandor.Geller at morganstanley.com> wrote:

> Hello,
>
> On Fri, May 13, 2011 at 7:43 PM, Zeek Anow <zeekstern at gmail.com> wrote:
> >
> > Something really wrong with syslog-ng or my config. I'm dropping way too
> > many packets.
> > I will admit that my configuration is probably really a large part of the
> > problem
> > and would appreciate it if someone could take a look at it and offer some
> > suggestions.
> >  There is another thread going about a similar problem on a similar
> > platform.
> >
> >  We recently upgraded to Solaris 10 from Solaris 9 and I don't recall us
> > dropping that
> > many packets before. And we also upgraded from a very older Sylog-ng
> version
> > to 3.1.2.
> > I am basing the dropped packets on the udp stats, not syslog-ng stats.
> > Syslog-ng stats has NO dropped packets.
>
> This implies that syslog-ng couldn't read the messages from the UDP
> socket so the kernel dropped incoming logs to the floor.
>
> > UDP     udpInDatagrams      -4599313    udpInErrors         -     0
> >         udpOutDatagrams     -  3421     udpOutErrors        -     0
> >         tcpInErrs           -     0     udpNoPorts          -2587612
> >         udpInCksumErrs      -     0     udpInOverflows      -95806254
> >
>
> ...
>
> > Below is a very small sampling of our syslog-ng.conf. We are filtering on
> > about
> > 1400 devices most of which are Firewalls and routers. The IPs in the
> > following
> > sample have been made up.
>
> 1400 *distinct* log sources like TCP streams? A single-threaded
> syslog-ng instance might not scale for such amount of log sources. An
> option would be running multiple syslog-ng instances listening on
> different ports and configure clients to use different ports sharing
> the load between the syslog-ng instances.
>
> > One of my questions is "Does the number of devices we are filtering on
> make
> > a difference? (1400)"
> > We have several sites and use just one version of the syslogng.conf file.
> It
> > is
> > a lot easier to maintain one copy:))
> >
> > Also notice the format:  ("^10\.123\.10\.133$") for the filters. All 1400
> > are in that format.
> >  I was hoping this would help a little but don't really know for sure:))
>
> Could you make your filters hierarchical? Using the netmask() filter
> and nested log statements you could reduce the number of filter
> evaluations. For example
>
> filter f_tenten {
>  netmask("10.10/16";)
> };
>
> filter tentenone {
>  netmask("10.10.1/24");
> };
>
> filter tententwo {
>  netmask("10.10.2/24");
> };
>
> filter f_teneleven {
>  netmask("10.11/16";)
> };
>
> ...
>
> and later
>
> log {
>  # first big network
>  source(...);
>  filter(f_tenten);
>  log {
>    filter(f_tentenone);
>    destination(...);
>  };
>  log {
>    filter(tententwo);
>    destination(...);
>  };
>  flags(final);
> };
>
> log {
>  # second big network
>  source(...);
>  filter(f_teneleven);
>  ...
> };
>
> and so on, organizing filters into tree / forest hierarchy instead of
> evaluating all filters one by one for all messages. This way logs
> coming from 10.10/16 will get processed in the first log block. Logs
> originating from other networks will trigger the evaluation of the
> f_tenten filter and as it will give a false result none of the
> embedded log statements (including their filters) will process the
> log. So the logs will get processed in the next log section (with the
> second big network comment).
>
> Using flags(final) is very important here, it tells syslog-ng to do
> not process more log{} sections for the given log message, otherwise
> all further log sections will process the message. If you want to log
> a message to multiple places then just add the additional destinations
> to the proper log{} block.
>
> Regards,
>
> Sandor
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110514/dbeab680/attachment.htm