I'm getting upwards of 15k mps no problem on older hardware with rmem default/max = 124928/131071, so your numbers sound pretty off. One thing to do would be to make sure syslog-ng isn't blocking on DNS all the time. A great sanity check is to use nc -l 514 -u > /some/out/file to prove that your kernel settings can keep up. Send a bunch of packets with loggen, then do a wc -l /some/out/file to check how many were received. If that's looking good, I recommend running tcpdump/wireshark to find the rate of DNS lookups from the box. If the cache is working, there should hardly be any after a minute or two. A full answer would require seeing the values of your output templates. On Fri, Oct 15, 2010 at 3:39 PM, Lars Kellogg-Stedman <lars@oddbit.com> wrote:
Hello all,
I'm deploying syslog-ng 3.0.8 on a quad-core 2.4Ghz system with 4GB of memory. Using stock kernel settings (e.g., without adjusting net.core.rmem_default), we're not able to handle much more than 100 messages/second (generated from a remote host using the "loggen" tool). At 500 msg/sec (-r 500), we see about 50% loss, and at 1000 msg/sec, we see closer to 60% packet loss.
Our configuration looks approximately like this (template definitions elided for brevity):
options { time_reap(30); mark_freq(10); keep_hostname(yes); use_fqdn(yes); dns_cache(2000); dns_cache_expire(86400); };
source s_network { udp(); tcp(port(514)); };
destination d_syslog { file("/srv/syslog/bydate/$YEAR-$MONTH-$DAY/messages" template(t_daily_log) create_dirs(yes) ); file("/srv/syslog/byhost/$FULLHOST_FROM/$YEAR-$MONTH-$DAY" template(t_host_log) create_dirs(yes) ); };
log { source(s_network); destination(d_syslog); };
I didn't think these message rates were terribly high, so I was surprised at the loss. We've confirmed that the loss is entirely between the kernel and the application -- using wireshark, we've verified that all of the packets are arriving at the host, and using this:
awk '{print}' /inet/udp/514/0/0 > out
Our packet loss is < 1%.
If I raise the rmem settings like this:
net.core.rmem_default = 512000 net.core.rmem_max = 1024000
Then it looks like I can support messages rates around 1000 msgs/sec. If I try with 2000 msgs/sec, the loss rates jumps up again (to around 30%).
Do these numbers make sense? This is an unloaded server. The only log traffic hitting this system is from my loggen runs. The filesystem is ext3 on top of a hardware RAID5 array. I've tried fiddling with some of the syslog-ng global options (e.g., flush_lines(), log_fetch_limit()), but without having much impact on performance.
I would appreciate any help you can send our way. Thanks!
-- Lars ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html