[syslog-ng] Tips for handling large message load

[Gabor Nagy (gnagy)]

Hi Mark,

I have some tips:

  *   disable flow-control for UPD sources. Flow-control will not prevent message loss in case of UDP protocol, hence the only thing it can cause is that your syslog-ng source will not read messages from the kernel buffer and then packet drops can happen.
  *   max-connections() does not scale UDP sources, there is no need to set it.
  *   flush-lines() are not affecting network destinations (network(), syslog(), unix-*(), drivers) performance, however the admin guide is not clear about this. We will fix this.
  *   log_iw_size() option is related to flow-control, no need to set it.
  *   set log-fetch-limit() to higher so you source can read more messages in a poll. The default value is 10, so trying out with e.g. 100 is a good start.
  *   you also need to adjust destination buffer's size with log-fifo-size().
You need to make sure that you log-fifo-size can hold enough messages without packet drop, e.g. if you can close to the max kernel buffer you configured.

We have a chapter about configuring syslog-ng parameters, it mostly focuses on flow-control:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/54#TOPIC-1209277

Regards,
Gabor
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov>
Sent: Monday, July 29, 2019 16:59
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Tips for handling large message load

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


I have several Splunk log aggregators that gets thousands of messages per second but we are seeing issues with dropping messages from UDP sources.

I've read the section in the docs about handling large message load and we've made many of those changes.  Do you have any other suggestions to improve performance?

We are using flow control.   We have made the following sysctl changes:
- net.core.rmem_max = 268435456
- net.core.netdev_max_backlog = 2000

We have increased flush-lines to 100

We are also looking to increase the initial window size below and max connections as well as the so_rcvbuf.

  network(port(514) transport("tcp") max-connections(100) log_iw_size(10000) flags(syslog-protocol));
  network(port(514) transport("udp") max-connections(100) log_iw_size(10000) flags(syslog-protocol));
  network(ip(0.0.0.0) transport("tls") port(10514) max-connections(100) log_iw_size(10000) flags(syslog-protocol)

I'm currently showing about 50 TCP connections and almost 1000 UDP connections, though it's early on Monday morning so it is likely to increase as the day goes on.  Most of the log messages are coming from the VPNs (hence so many UDP connections).

The servers are quite beefy with 64GB of RAM and 24 Xeon cores @ 2.4GHz (Dell PowerEdge R530).  We are using two separate network interfaces one for TCP and the other for UDP.

I'd appreciate any suggestions on how to further increase performance, also, any general rules or calculations I can use to determine optimal values for these parameters from available system metrics would be very helpful.

Thanks,
-Mark
______________________________________________________________________________
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=AWkyZjqIhSW2OU7%2F2gFUL3DBfIAmaJ5Rb8sWC87MOho%3D&reserved=0
Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=khjr%2FWBVx1V4lR9bB67i5KX%2BxTGWYinQIa4utv8LNO8%3D&reserved=0
FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=7mevZAo06Ry0el0BAstkUbo3ql7zP2FV7vw5gO2inl8%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190730/49932074/attachment.html>