Hi Lars, try increasing your UDP receive buffers as outlined here:<div><a href="http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers">http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers</a></div>
<div><a href="http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers"></a><br clear="all">______________________________________________________________ <br><br>Clayton Dukes<br>______________________________________________________________<br>
<br><br><div class="gmail_quote">On Fri, Oct 15, 2010 at 4:39 PM, Lars Kellogg-Stedman <span dir="ltr"><<a href="mailto:lars@oddbit.com">lars@oddbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hello all,<br>
<br>
I'm deploying syslog-ng 3.0.8 on a quad-core 2.4Ghz system with 4GB of<br>
memory. Using stock kernel settings (e.g., without adjusting<br>
net.core.rmem_default), we're not able to handle much more than 100<br>
messages/second (generated from a remote host using the "loggen"<br>
tool). At 500 msg/sec (-r 500), we see about 50% loss, and at 1000<br>
msg/sec, we see closer to 60% packet loss.<br>
<br>
Our configuration looks approximately like this (template definitions<br>
elided for brevity):<br>
<br>
options {<br>
time_reap(30);<br>
mark_freq(10);<br>
keep_hostname(yes);<br>
use_fqdn(yes);<br>
dns_cache(2000);<br>
dns_cache_expire(86400);<br>
};<br>
<br>
source s_network {<br>
udp();<br>
tcp(port(514));<br>
};<br>
<br>
destination d_syslog {<br>
file("/srv/syslog/bydate/$YEAR-$MONTH-$DAY/messages"<br>
template(t_daily_log)<br>
create_dirs(yes)<br>
);<br>
file("/srv/syslog/byhost/$FULLHOST_FROM/$YEAR-$MONTH-$DAY"<br>
template(t_host_log)<br>
create_dirs(yes)<br>
);<br>
};<br>
<br>
log {<br>
source(s_network);<br>
destination(d_syslog);<br>
};<br>
<br>
I didn't think these message rates were terribly high, so I was<br>
surprised at the loss. We've confirmed that the loss is entirely<br>
between the kernel and the application -- using wireshark, we've<br>
verified that all of the packets are arriving at the host, and using<br>
this:<br>
<br>
awk '{print}' /inet/udp/514/0/0 > out<br>
<br>
Our packet loss is < 1%.<br>
<br>
If I raise the rmem settings like this:<br>
<br>
net.core.rmem_default = 512000<br>
net.core.rmem_max = 1024000<br>
<br>
Then it looks like I can support messages rates around 1000 msgs/sec.<br>
If I try with 2000 msgs/sec, the loss rates jumps up again (to around<br>
30%).<br>
<br>
Do these numbers make sense? This is an unloaded server. The only<br>
log traffic hitting this system is from my loggen runs. The<br>
filesystem is ext3 on top of a hardware RAID5 array. I've tried<br>
fiddling with some of the syslog-ng global options (e.g.,<br>
flush_lines(), log_fetch_limit()), but without having much impact on<br>
performance.<br>
<br>
I would appreciate any help you can send our way. Thanks!<br>
<br>
-- Lars<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</blockquote></div><br></div>