Hello Lars, In my own experience coding UDP applications I could handle upwards of 3000 messages per second. However I was not trying to output them to a file. What network adapter and Ethernet driver does the machine use? Which operating system? Could you try a different operating system as a process of elimination (maybe switch between Linux, BSD, and Solaris)? Did you try tweaking: log_fetch_limit() so_rcvbuf() so_sndbuf() http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid... I have noticed in my own code that SO_RCVBUF makes a HUGE difference. Do you know for sure that when 'loggen' says it's creating X msgs/sec, that this reading is actually accurate? Can you profile syslog-ng using oprofile or the various built in profiling and debugging options in its configure script? Can you check if the CFLAGS look good? -O2 and -Os are usually the best, for me -O3 seems to slow things down. Also check the arch and tune settings. Consider trying it with the Intel C Compiler to see if this helps. Let's try to get some more details about your platform and identify the bottlenecks so we can get to the details quickly and avoid guessing too much. Matthew. On Fri, Oct 15, 2010 at 04:39:54PM -0400, Lars Kellogg-Stedman wrote:
Hello all,
I'm deploying syslog-ng 3.0.8 on a quad-core 2.4Ghz system with 4GB of memory. Using stock kernel settings (e.g., without adjusting net.core.rmem_default), we're not able to handle much more than 100 messages/second (generated from a remote host using the "loggen" tool). At 500 msg/sec (-r 500), we see about 50% loss, and at 1000 msg/sec, we see closer to 60% packet loss.
Our configuration looks approximately like this (template definitions elided for brevity):
options { time_reap(30); mark_freq(10); keep_hostname(yes); use_fqdn(yes); dns_cache(2000); dns_cache_expire(86400); };
source s_network { udp(); tcp(port(514)); };
destination d_syslog { file("/srv/syslog/bydate/$YEAR-$MONTH-$DAY/messages" template(t_daily_log) create_dirs(yes) ); file("/srv/syslog/byhost/$FULLHOST_FROM/$YEAR-$MONTH-$DAY" template(t_host_log) create_dirs(yes) ); };
log { source(s_network); destination(d_syslog); };
I didn't think these message rates were terribly high, so I was surprised at the loss. We've confirmed that the loss is entirely between the kernel and the application -- using wireshark, we've verified that all of the packets are arriving at the host, and using this:
awk '{print}' /inet/udp/514/0/0 > out
Our packet loss is < 1%.
If I raise the rmem settings like this:
net.core.rmem_default = 512000 net.core.rmem_max = 1024000
Then it looks like I can support messages rates around 1000 msgs/sec. If I try with 2000 msgs/sec, the loss rates jumps up again (to around 30%).
Do these numbers make sense? This is an unloaded server. The only log traffic hitting this system is from my loggen runs. The filesystem is ext3 on top of a hardware RAID5 array. I've tried fiddling with some of the syslog-ng global options (e.g., flush_lines(), log_fetch_limit()), but without having much impact on performance.
I would appreciate any help you can send our way. Thanks!
-- Lars ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html