Thanks everyone for the quick feedback!

I am asking our vmware folks to take a look from the "outside" for things that might help (we may be having a shared resource issue for example)

I am working on a test environment to look into further tuning with kernel settings.

I am also looking at application changes (since this is seen on both a syslog-ng "collector" and a splunk "forwarder" I am looking at the kernel / OS stuff first).

One possibility is that we could stand up a load balancer in front and spread the load across multiple boxes. 

Thanks - If / when I have any brilliant success I will pass it along :-)

Jim


On Wed, Jan 11, 2017 at 8:06 AM, Clayton Dukes <cdukes@logzilla.net> wrote:

Jim,

You can try some of the steps outlined here:

http://demo.logzilla.net/help/performance_tuning/udp_buffer_tuning

 

Since these are VM's, you may also want to look at the type of NIC you are using for the VM, for example:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2019944

 

And of course, make sure the VMware drivers are installed in the OS.

 

 

From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Jim Hendrick <james.r.hendrick@gmail.com>
Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Date: Tuesday, January 10, 2017 at 4:20 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Subject: [syslog-ng] udp tuning - where to look.

 

Hi,

 

  I am working an issue with UDP drops in a high EPS rate environment (several thousand steady) and struggling with making any changes in the loss rate. (according to netstat -su | grep error).

 

  This is happening on two nodes, (one running an older syslog-ng 3.2 OSE, the other running a splunk forwarder) but I don't actually think it is making it out of the kernel to either application.

 

  On the syslog-ng side syslog-ng-ctl stats shows *no* drops at all.

 

  Increasing net.core.rmem_max and so_rcvbuf together all the way to 64 MB did not seem to make any significant difference.

 

  This is a RHEL 6 box with 16 GB and 4 cores (virtual - running in an ESX environment)

 

  Are there other parameters, things I should be looking at?

 

Thanks,

 

Jim

 

 

 

 

 

 


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq