[syslog-ng] High throughput UDP logging configuration.

Evan Rempel erempel at uvic.ca
Thu May 28 13:45:51 UTC 2020

I know. There is a ton of information online about this. It shouldn't be 
that difficult, but I'm still having problems.

I have one device with a very high logging rate. Approaching 15K msg/sec 

I am running on a 2 socket 8 core/CPU system (Intel(R) Xeon(R) CPU 
E5-2670 0 @ 2.60GHz). Not the latest and greatest, but still fairly 
quick. All of the storage is on SSD.

What I have configured

1. OS net.core.rmem_max = 536870912
2. so-reuseport(1) - and 8 sources on the UDP port
3. so_rcvbuf(67108864) - all 9 udp sources.
4. log_iw_size(2M)
5. log_fetch_limit(20k)
6. log_fifo_size(4M)

Flow control is NOT enabled.

Monitoring the queued messages inside syslog-ng and they remain near zero.

Monitoring udp buffer queues in the OS shows that one stream is still 
overwhelming syslog-ng's ability to read messages.

[~]$ sudo netstat -unlp|egrep -e 'PID|syslog'
Proto   Recv-Q Send-Q Local Address     Foreign Addr  State PID/Program 
udp          0      0* 15591/syslog-ng
udp          0      0* 15591/syslog-ng
udp          0      0* 15591/syslog-ng
udp          0      0* 15591/syslog-ng
udp          0      0* 15591/syslog-ng
udp  134216320      0* 15591/syslog-ng
udp          0      0* 15591/syslog-ng
udp          0      0* 15591/syslog-ng

A few things of note.

1. The OS UDP buffer seems to be 128MB in size and the so_rcvbuf 
configured ins 64M in size. Is that because the syslog-ng configuration 
of so_rcvbuf is in characters but the OS buffer is in bytes?

2. In a 1 second interval "cat /proc/net/udp" shows that one UDP stream 
dropped 6283 packets. So I'm continually dropping approx 50% of the UDP 

3. Increasing the log_iw_size or the log_iw_size actually seems to make 
things worse.

All suggestions that help me understand this and help to minimize the 
drops are welcome.

Evan Rempel

More information about the syslog-ng mailing list