High throughput UDP logging configuration.
I know. There is a ton of information online about this. It shouldn't be that difficult, but I'm still having problems. I have one device with a very high logging rate. Approaching 15K msg/sec (7.5MB/sec). I am running on a 2 socket 8 core/CPU system (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz). Not the latest and greatest, but still fairly quick. All of the storage is on SSD. What I have configured 1. OS net.core.rmem_max = 536870912 2. so-reuseport(1) - and 8 sources on the UDP port 3. so_rcvbuf(67108864) - all 9 udp sources. 4. log_iw_size(2M) 5. log_fetch_limit(20k) 6. log_fifo_size(4M) Flow control is NOT enabled. Monitoring the queued messages inside syslog-ng and they remain near zero. Monitoring udp buffer queues in the OS shows that one stream is still overwhelming syslog-ng's ability to read messages. [~]$ sudo netstat -unlp|egrep -e 'PID|syslog' Proto Recv-Q Send-Q Local Address Foreign Addr State PID/Program name udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 134216320 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng A few things of note. 1. The OS UDP buffer seems to be 128MB in size and the so_rcvbuf configured ins 64M in size. Is that because the syslog-ng configuration of so_rcvbuf is in characters but the OS buffer is in bytes? 2. In a 1 second interval "cat /proc/net/udp" shows that one UDP stream dropped 6283 packets. So I'm continually dropping approx 50% of the UDP stream. 3. Increasing the log_iw_size or the log_iw_size actually seems to make things worse. All suggestions that help me understand this and help to minimize the drops are welcome. -- Evan Rempel
Hi,
1. The OS UDP buffer seems to be 128MB in size and the so_rcvbuf configured ins 64M in size. Is that because the syslog-ng configuration of so_rcvbuf is in characters but the OS buffer is in bytes?
This is because the kernel doubles the value set by syslog-ng (to allow space for bookkeeping overhead), and this doubled value is returned by getsockopt(2) and other tools.
3. Increasing the log_iw_size or the log_iw_size actually seems to make things worse.
These 2 values already seem high enough. Disabling flow-control is also a good idea IMO, when using UDP sources.
All suggestions that help me understand this and help to minimize the drops are welcome.
Could you share how incoming packets are distributed across the 8 sockets? The default SO_REUSEPORT mechanism distributes packets based on the hash of (peer IP address, port) and (local IP address, port), Hashing collision is also likely to happen [1], so if you encounter this problem, there are other possible resolutions. The commercial syslog-ng version has, for example, an udp-balancer() driver, that uses custom BPF programs to achieve an even distribution of packets. [1] https://blog.cloudflare.com/how-to-receive-a-million-packets/ -- László Várady
On 5/28/20 7:46 AM, László Várady (lvarady) wrote:
Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
Hi,
1. The OS UDP buffer seems to be 128MB in size and the so_rcvbuf configured ins 64M in size. Is that because the syslog-ng configuration of so_rcvbuf is in characters but the OS buffer is in bytes?
This is because the kernel doubles the value set by syslog-ng (to allow space for bookkeeping overhead), and this doubled value is returned by getsockopt(2) and other tools.
Thanks.
3. Increasing the log_iw_size or the log_iw_size actually seems to make things worse.
These 2 values already seem high enough. Disabling flow-control is also a good idea IMO, when using UDP sources.
Flow control is completely disabled. Or more precisely unspecified. It occurs to me that perhaps the default is enabled?
All suggestions that help me understand this and help to minimize the drops are welcome.
Could you share how incoming packets are distributed across the 8 sockets?
The default SO_REUSEPORT mechanism distributes packets based on the hash of (peer IP address, port) and (local IP address, port), Hashing collision is also likely to happen [1], so if you encounter this problem, there are other possible resolutions. The commercial syslog-ng version has, for example, an udp-balancer() driver, that uses custom BPF programs to achieve an even distribution of packets.
It is a single device, single IP, single port, so there is just one socket that is overwhelmed. This will probably be the same failure to process fast enough if we move to a TCP transport.
[1] https://blog.cloudflare.com/how-to-receive-a-million-packets/
I will read through this blog. -- Evan Rempel
Hello,
It is a single device, single IP, single port, so there is just one socket that is overwhelmed. This will probably be the same failure to process fast enough if we move to a TCP transport.
Yeah, a (single IP, single port) sender won't scale across CPU cores with the default SO_REUSEPORT distribution. However, 15K msg/sec should be handled without problems. For example, we've measured 170K msg/sec (70 MB/sec) rates with a single listener using so_rcvbuf(1M) in our performance testing environment. https://github.com/syslog-ng/syslog-ng/issues/3309#issuecomment-635217670 might be interesting to you. -- László Várady
participants (2)
-
Evan Rempel
-
László Várady (lvarady)