[syslog-ng] Solaris 10 UDP overflows, message drops

[Mishou Michael]

All,

I've done a lot of reading, and I can't figure out what I can do to this
config in order to fix the UDP drops due to udpInOverflows on netstat
-s.  Here are some statistics relating to the amount of traffic we
receive via syslog-ng, it's pretty busy but in reading I'm finding that
some folks are doing much more.  These stats are based on a ~30 second
window of traffic during peak times, but variance due to time is not so
much in our environment.  I used tcpdump with a bpf to capture only
inbound udp/514, so this is what the interface is seeing in the way of
syslog.

Elapsed:		00:00:34
Packets:		200000
Avg. packets/sec:	5836.546
Avg. packet size:	303.182 bytes
Bytes:		60636477
Avg. bytes/sec:	1769537.884
Avg. MBit/sec:	14.156

So, about 6k messages per second.  Here are the drop numbers over a time
sample (done right after a process restart, you can see the buffer takes
a moment to fill up [64 MB so_rcvbuf]):

# while true; do echo -en "$(date) :: "; netstat -s | grep
udpInOverflows | head -n 1 | sed 's|.*=||'; sleep 10; done
Fri Apr 15 14:12:46 GMT 2011 :: 472517477
Fri Apr 15 14:12:56 GMT 2011 :: 472517477
Fri Apr 15 14:13:06 GMT 2011 :: 472517477
Fri Apr 15 14:13:16 GMT 2011 :: 472517477
Fri Apr 15 14:13:26 GMT 2011 :: 472543152
Fri Apr 15 14:13:36 GMT 2011 :: 472592800
Fri Apr 15 14:13:46 GMT 2011 :: 472638848
Fri Apr 15 14:13:56 GMT 2011 :: 472684407

So that's about 5k overflows a second, which jives with our
calculations, suggesting we're getting only ~10% of our messages logged
to disk.

I inherited a config with _very_ many filter statements, but have
decided to cut all that out to see if my performance problems in the way
of udp drops continue (they do).  I've attached a sanitized config to
this message, all the stuff here concerns this config running (even
though I thought eliminating the filters would really help, it didn't).

We're running Solaris 10 SPARC.  The syslog-ng version is:

# /usr/local/sbin/syslog-ng -V
syslog-ng 3.1.2
Installer-Version: 3.1.2
Revision:
ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainli
ne--3.1#master#8bf13c304b6ab5fc1a372b49d55c78370efe14ca
Compile-Date: Oct 25 2010 23:56:18
Enable-Threads: off
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-Sun-STREAMS: on
Enable-Sun-Door: on
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: off
Enable-SSL: on
Enable-SQL: off
Enable-Linux-Caps: off
Enable-Pcre: on

The following options are set for the OS:

# ndd /dev/udp udp_max_buf
1073741824
# ndd /dev/udp udp_recv_hiwat
65536

Some options lines from the config based on what I've seen:

* note the TCP stuff can be safely ignored, it's legacy from some
testing but isn't currently seeing traffic
* all 3 udp sources set with so_rcvbuf(67108864) (64 MB)

options { # things I've changed/tweaked
          flush_lines(1000);
          flush_timeout(20);
          log_fifo_size (67108864);
          log_msg_size(8192);
          chain_hostnames(yes);
          # end my changes
        <snip>
        };

So I'm totally stumped.  I can set the buffers with so_rcvbuf() to 1 GB,
it still doesn't matter, they eventually fill up and I start losing
packets.  I'm hoping that someone can point me to some tweaks I can do
to get the numbers of drops down or eliminated.  Is it unreasonable to
expect to be able to process this many messages per second via UDP?
Maybe that's the problem.  I might experiment some with default syslog
to see if it can write this many messages without drops...this doesn't
seem like an insane amount of traffic.  But perhaps my expectations are
unrealistic, that's what I'm hoping someone can tell me.

Regards,

--Mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: syslog-ng.conf.sanitized
Type: application/octet-stream
Size: 5203 bytes
Desc: syslog-ng.conf.sanitized
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110415/ad7ee56d/attachment.obj