On Sat, 2009-05-30 at 10:57 -0400, Jan Schaumann wrote:
Hello,
I have a FreeBSD 6.2 (amd64) host where I'd like to replace the stock syslogd with syslog-ng (3.0.2). This host receives a lot of syslog messages per second from a large number of clients via UDP.
The stock syslogd configuration is trivial:
*.* /var/log/all
This host currently drops about 2-4% of all UDP packets, syslog takes about 50-65% of one CPU.
A drop-in replacement configuration using syslog-ng:
options { create_dirs(yes); use_dns(no); };
template t_default { template("${DATE} <${FACILITY}.${PRIORITY}> ${HOST} ${MSG}\n"); };
source s_standard { file("/dev/klog"); internal(); udp(); unix-dgram("/var/run/log"); };
destination d_all { file("/var/log/all" template(t_default) ); };
# *.* /var/log/all log { source(s_standard); destination(d_all); };
syslog-ng uses about 90% of one CPU and drops between 15% and 20% of UDP packets (and, based on traffic patterns and logfile size, concurrent logfile rotation etc. even as high as 30%).
Hmm.. one possible problem is that syslog-ng wakes up too often processes a small number of messages and goes back to sleep. Since the poll iteration has its overhead, this might add up to be significant. You could perhaps play with time_sleep(), I'd go for 30msecs which would limit syslog-ng to wake up at most 30 times per second. Then, make sure that you actually have a large enough UDP receive buffer. so_rcvbuf() might not be enough, as systems usually add further limits on the maximum per-socket receive buffer size. On Linux, it is /proc/sys/net/core/rmem_{max,default} Here's what I've found using Google: http://www.29west.com/docs/THPM/udp-buffer-sizing.html sysctl -w kern.ipc.maxsockbuf=<number> I would increase the size of the receive buffer to at least 3-5 seconds worth of messages, so given you have 15k/sec, you'd need about 22MB of receive buffer space.
I've tried a number of things to improve this performance, including:
log_fetch_limit(100); log_iw_size(10000); flush_lines(100000); flush_timeout(10);
fetch_limit() might be related, if you have only a small number of sources, you could increase that, but don't forget to adjust the destination window size, as described in the documentation: http://wwwen.balabit/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html flush stuff should not matter that match.
in the global options and
log_fifo_size(100000)
in the destination definition
with
flags(flow-control)
in the log definition.
flow-control is not a NOP for udp() sources as Sandor said, if the flow-control kicks in, syslog-ng will completely stop receiving udp() messages altogether, basically making the kernel drop them. flow-control for destination files is however basically a nop as destination files are _always_ writable.
The best I was able to get with these numbers was 5-7% of UDP drops (ie still double of what the stock syslogd drops).
I also tried adjusting "so_rcvbuf" for UDP with no noticable difference.
Please check the kernel parameters.
Now consider that I did not do any sysctl tuning, as those should equally influence the stock syslog and I'm trying to sort out why one performs so significantly better than the other.
syslog-ng core can do about 130k msg/sec without writing things to files, and about 70k/sec if you have a single destination file. however it might have a latency that causes the udp() receive buffer to fill up. If you carefully size your udp() receive buffer you can probably achieve no message losses for about 15k msg/sec. I'd guess you'd be losing messages for sure if we're talking over 20k msg/sec
Leaving aside any of the things I can do with syslog-ng further down the road (such as filtering and intentionally dropping certain messages), what can I do to get syslog-ng up to the same performance as the stock syslogd?
Many thanks in advance for any pointers and help.
-- Bazsi