[syslog-ng] Tips for handling large message load

[Faine, Mark R. (MSFC-IS40)[NICS]]

I have several Splunk log aggregators that gets thousands of messages per second but we are seeing issues with dropping messages from UDP sources.
  
I've read the section in the docs about handling large message load and we've made many of those changes.  Do you have any other suggestions to improve performance?

We are using flow control.   We have made the following sysctl changes:
- net.core.rmem_max = 268435456
- net.core.netdev_max_backlog = 2000

We have increased flush-lines to 100

We are also looking to increase the initial window size below and max connections as well as the so_rcvbuf. 

  network(port(514) transport("tcp") max-connections(100) log_iw_size(10000) flags(syslog-protocol));
  network(port(514) transport("udp") max-connections(100) log_iw_size(10000) flags(syslog-protocol));
  network(ip(0.0.0.0) transport("tls") port(10514) max-connections(100) log_iw_size(10000) flags(syslog-protocol)

I'm currently showing about 50 TCP connections and almost 1000 UDP connections, though it's early on Monday morning so it is likely to increase as the day goes on.  Most of the log messages are coming from the VPNs (hence so many UDP connections).

The servers are quite beefy with 64GB of RAM and 24 Xeon cores @ 2.4GHz (Dell PowerEdge R530).  We are using two separate network interfaces one for TCP and the other for UDP.  

I'd appreciate any suggestions on how to further increase performance, also, any general rules or calculations I can use to determine optimal values for these parameters from available system metrics would be very helpful. 

Thanks,
-Mark