[syslog-ng] UDP drops with syslog-ng 3.24.1-1

[László Várady (lvarady)]

Hi,



  *   How does the so-rcvbuf of the kernel (rmem_max) correlate to the so-rcvbuf in syslog-ng, especially when using so-reuseport.  Should the rmem_max be the sum of all the so-rcvbuf settings in syslog-ng?  Is rmem_max a per CPU buffer?  In general, how should these variables be scaled in unison?  In our case, our so-rcvbuf is 441,326,592 across 4 so-reuseports, while our rmem_max is 268,435,456.

The so-rcvbuf() option of syslog-ng sets the SO_RCVBUF socket options, so the two are the same (syslog-ng sets the kernel buffer size). When using so-reuseport(), each UDP source consists of a separate socket. so-rcvbuf() can be set separately for each socket if necessary.
Note that the kernel doubles this value (to allow space for bookkeeping overhead).
We have a warning in case the value is not accepted by the kernel:
    The kernel refused to set the receive buffer (SO_RCVBUF) to the requested size, you probably need to adjust buffer related kernel parameters

net.core.rmem_max specifies the maximum configurable value, but it is per-socket limit, so it does NOT have to be the sum of all sockets.
The default value can be also set with net.core.rmem_default, but you can override this with the so-rcvbuf() syslog-ng option up to the rmem_max limit.


  *   We record all of our syslog-ng stats at 5 minute intervals with syslog-ng-ctl.  If this never shows any drops, would that imply that this is more of an issue with kernel buffers than syslog-ng itself?

In case of UDP, you can't avoid message loss, so it does not really matter where messages are dropped. If you configured flags(flow-control) in any of the UDP log paths, syslog-ng would not drop messages, but the kernel will definitely do it instead. I don't recommend using flow-control for UDP sources. Without flow control, syslog-ng reads messages as fast as it can and drops new messages when a destination becomes unavailable/too slow to process the throughput. This can be monitored with syslog-ng-ctl (dropped counters).
Kernel message drops are also possible in this scenario, but it means that the given syslog-ng config/environment is unable to process the incoming traffic fast enough, so something has to be adjusted or scaled.


  *   Is there a syslog-ng command that will show its queues, to see how those are being utilized?

All destination queues can be monitored with syslog-ng-ctl stats. Those counters end with the "queued" suffix, for example:
dst.network;d_ise#1;tcp,127.0.0.1:5555;a;queued;1

--
László Várady
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200520/5008be55/attachment-0001.html>