[syslog-ng] udp drops

[Balazs Scheidler]

On Sat, 2009-05-30 at 10:57 -0400, Jan Schaumann wrote:
> Hello,
> 
> I have a FreeBSD 6.2 (amd64) host where I'd like to replace the stock
> syslogd with syslog-ng (3.0.2).  This host receives a lot of syslog
> messages per second from a large number of clients via UDP.
> 
> The stock syslogd configuration is trivial:
> 
> *.*	/var/log/all
> 
> This host currently drops about 2-4% of all UDP packets, syslog takes
> about 50-65% of one CPU.
> 
> A drop-in replacement configuration using syslog-ng:
> 
> options {
>         create_dirs(yes);
>         use_dns(no);
> };
> 
> template t_default {
>         template("${DATE} <${FACILITY}.${PRIORITY}> ${HOST} ${MSG}\n");
> };
> 
> source s_standard {  
>         file("/dev/klog");
>         internal();
>         udp();
>         unix-dgram("/var/run/log");
> };
> 
> destination d_all {
>         file("/var/log/all"
>                 template(t_default)
>                 );
> };
> 
> # *.*           /var/log/all
> log {   
>         source(s_standard);
>         destination(d_all);
> };
> 
> 
> syslog-ng uses about 90% of one CPU and drops between 15% and 20% of UDP
> packets (and, based on traffic patterns and logfile size, concurrent
> logfile rotation etc. even as high as 30%).

Hmm.. one possible problem is that syslog-ng wakes up too often
processes a small number of messages and goes back to sleep. Since the
poll iteration has its overhead, this might add up to be significant.

You could perhaps play with time_sleep(), I'd go for 30msecs which would
limit syslog-ng to wake up at most 30 times per second.

Then, make sure that you actually have a large enough UDP receive
buffer. so_rcvbuf() might not be enough, as systems usually add further
limits on the maximum per-socket receive buffer size.

On Linux, it is /proc/sys/net/core/rmem_{max,default}

Here's what I've found using Google:

http://www.29west.com/docs/THPM/udp-buffer-sizing.html

sysctl -w kern.ipc.maxsockbuf=<number>

I would increase the size of the receive buffer to at least 3-5 seconds
worth of messages, so given you have 15k/sec, you'd need about 22MB of
receive buffer space.

> 
> 
> I've tried a number of things to improve this performance, including:
> 
>        log_fetch_limit(100);
>        log_iw_size(10000);
>        flush_lines(100000);
>        flush_timeout(10);

fetch_limit() might be related, if you have only a small number of
sources, you could increase that, but don't forget to adjust the
destination window size, as described in the documentation:

http://wwwen.balabit/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html

flush stuff should not matter that match.

> 
> in the global options and
> 
> 	log_fifo_size(100000)
> 
> in the destination definition
> 
> with
> 
> 	flags(flow-control)
> 
> in the log definition.

flow-control is not a NOP for udp() sources as Sandor said, if the
flow-control kicks in, syslog-ng will completely stop receiving udp()
messages altogether, basically making the kernel drop them.

flow-control for destination files is however basically a nop as
destination files are _always_ writable.

> 
> The best I was able to get with these numbers was 5-7% of UDP drops (ie
> still double of what the stock syslogd drops).
> 
> I also tried adjusting "so_rcvbuf" for UDP with no noticable difference.

Please check the kernel parameters.

> 
> Now consider that I did not do any sysctl tuning, as those should
> equally influence the stock syslog and I'm trying to sort out why one
> performs so significantly better than the other.

syslog-ng core can do about 130k msg/sec without writing things to
files, and about 70k/sec if you have a single destination file. however
it might have a latency that causes the udp() receive buffer to fill up.
If you carefully size your udp() receive buffer you can probably achieve
no message losses for about 15k msg/sec.

I'd guess you'd be losing messages for sure if we're talking over 20k
msg/sec

> 
> Leaving aside any of the things I can do with syslog-ng further down the
> road (such as filtering and intentionally dropping certain messages),
> what can I do to get syslog-ng up to the same performance as the stock
> syslogd?
> 
> Many thanks in advance for any pointers and help.

-- 
Bazsi