<div dir="ltr">Thanks everyone for the quick feedback!<div><br></div><div>I am asking our vmware folks to take a look from the "outside" for things that might help (we may be having a shared resource issue for example)</div><div><br></div><div>I am working on a test environment to look into further tuning with kernel settings.</div><div><br></div><div>I am also looking at application changes (since this is seen on both a syslog-ng "collector" and a splunk "forwarder" I am looking at the kernel / OS stuff first).</div><div><br></div><div>One possibility is that we could stand up a load balancer in front and spread the load across multiple boxes. </div><div><br></div><div>Thanks - If / when I have any brilliant success I will pass it along :-)</div><div><br></div><div>Jim</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 11, 2017 at 8:06 AM, Clayton Dukes <span dir="ltr"><<a href="mailto:cdukes@logzilla.net" target="_blank">cdukes@logzilla.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_7403033261824717674WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Jim,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">You can try some of the steps outlined here:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><a href="http://demo.logzilla.net/help/performance_tuning/udp_buffer_tuning" target="_blank">http://demo.logzilla.net/help/<wbr>performance_tuning/udp_buffer_<wbr>tuning</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Since these are VM's, you may also want to look at the type of NIC you are using for the VM, for example:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><a href="https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2019944" target="_blank">https://kb.vmware.com/<wbr>selfservice/microsites/search.<wbr>do?language=en_US&cmd=<wbr>displayKC&externalId=2019944</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">And of course, make sure the VMware drivers are installed in the OS.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.<wbr>balabit.hu</a>> on behalf of Jim Hendrick <<a href="mailto:james.r.hendrick@gmail.com" target="_blank">james.r.hendrick@gmail.com</a>><br>
<b>Reply-To: </b>Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<b>Date: </b>Tuesday, January 10, 2017 at 4:20 PM<br>
<b>To: </b>Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject: </b>[syslog-ng] udp tuning - where to look.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Hi, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> I am working an issue with UDP drops in a high EPS rate environment (several thousand steady) and struggling with making any changes in the loss rate. (according to netstat -su | grep error).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> This is happening on two nodes, (one running an older syslog-ng 3.2 OSE, the other running a splunk forwarder) but I don't actually think it is making it out of the kernel to either application.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> On the syslog-ng side syslog-ng-ctl stats shows *no* drops at all.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Increasing net.core.rmem_max and <span style="font-size:11.0pt;font-family:Calibri">so_rcvbuf together all the way </span>to 64 MB did not seem to make any significant difference.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> This is a RHEL 6 box with 16 GB and 4 cores (virtual - running in an ESX environment)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Are there other parameters, things I should be looking at?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Jim<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>