[syslog-ng] High throughput UDP logging configuration.

[Evan Rempel]

I know. There is a ton of information online about this. It shouldn't be 
that difficult, but I'm still having problems.

I have one device with a very high logging rate. Approaching 15K msg/sec 
(7.5MB/sec).

I am running on a 2 socket 8 core/CPU system (Intel(R) Xeon(R) CPU 
E5-2670 0 @ 2.60GHz). Not the latest and greatest, but still fairly 
quick. All of the storage is on SSD.

What I have configured

1. OS net.core.rmem_max = 536870912
2. so-reuseport(1) - and 8 sources on the UDP port
3. so_rcvbuf(67108864) - all 9 udp sources.
4. log_iw_size(2M)
5. log_fetch_limit(20k)
6. log_fifo_size(4M)

Flow control is NOT enabled.

Monitoring the queued messages inside syslog-ng and they remain near zero.

Monitoring udp buffer queues in the OS shows that one stream is still 
overwhelming syslog-ng's ability to read messages.

[~]$ sudo netstat -unlp|egrep -e 'PID|syslog'
Proto   Recv-Q Send-Q Local Address     Foreign Addr  State PID/Program 
name
udp          0      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng
udp          0      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng
udp          0      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng
udp          0      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng
udp          0      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng
udp  134216320      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng
udp          0      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng
udp          0      0 11.22.33.44:514   0.0.0.0:* 15591/syslog-ng

A few things of note.

1. The OS UDP buffer seems to be 128MB in size and the so_rcvbuf 
configured ins 64M in size. Is that because the syslog-ng configuration 
of so_rcvbuf is in characters but the OS buffer is in bytes?

2. In a 1 second interval "cat /proc/net/udp" shows that one UDP stream 
dropped 6283 packets. So I'm continually dropping approx 50% of the UDP 
stream.

3. Increasing the log_iw_size or the log_iw_size actually seems to make 
things worse.


All suggestions that help me understand this and help to minimize the 
drops are welcome.


-- 
Evan Rempel