[syslog-ng] udp drops

[Jan Schaumann]

Hello,

I have a FreeBSD 6.2 (amd64) host where I'd like to replace the stock
syslogd with syslog-ng (3.0.2).  This host receives a lot of syslog
messages per second from a large number of clients via UDP.

The stock syslogd configuration is trivial:

*.*	/var/log/all

This host currently drops about 2-4% of all UDP packets, syslog takes
about 50-65% of one CPU.

A drop-in replacement configuration using syslog-ng:

options {
        create_dirs(yes);
        use_dns(no);
};

template t_default {
        template("${DATE} <${FACILITY}.${PRIORITY}> ${HOST} ${MSG}\n");
};

source s_standard {  
        file("/dev/klog");
        internal();
        udp();
        unix-dgram("/var/run/log");
};

destination d_all {
        file("/var/log/all"
                template(t_default)
                );
};

# *.*           /var/log/all
log {   
        source(s_standard);
        destination(d_all);
};


syslog-ng uses about 90% of one CPU and drops between 15% and 20% of UDP
packets (and, based on traffic patterns and logfile size, concurrent
logfile rotation etc. even as high as 30%).


I've tried a number of things to improve this performance, including:

       log_fetch_limit(100);
       log_iw_size(10000);
       flush_lines(100000);
       flush_timeout(10);

in the global options and

	log_fifo_size(100000)

in the destination definition

with

	flags(flow-control)

in the log definition.

The best I was able to get with these numbers was 5-7% of UDP drops (ie
still double of what the stock syslogd drops).

I also tried adjusting "so_rcvbuf" for UDP with no noticable difference.

Now consider that I did not do any sysctl tuning, as those should
equally influence the stock syslog and I'm trying to sort out why one
performs so significantly better than the other.

Leaving aside any of the things I can do with syslog-ng further down the
road (such as filtering and intentionally dropping certain messages),
what can I do to get syslog-ng up to the same performance as the stock
syslogd?

Many thanks in advance for any pointers and help.
-Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090530/f0a36f80/attachment.pgp