<html><head></head><body><div class="ydp791e652fyahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false">We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Here are some relevant configuration items:</div><div dir="ltr" data-setdir="false">2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running <span><span style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">syslog-ng-3.24.1-1.el7.x86_64.</span></span></div><div dir="ltr" data-setdir="false"><div><div>net.core.rmem_default = 212992</div><div>net.core.rmem_max = 268435456</div></div><div dir="ltr" data-setdir="false"><br></div><span>log_fifo_size(268435456);</span><br></div><div dir="ltr" data-setdir="false"><span><br></span></div><div dir="ltr" data-setdir="false"><div><div style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">source s_network {</div><div style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));</div><div style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));</div><div style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));</div><div style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));</div><div style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));</div><div style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">};</div></div></div><div dir="ltr" data-setdir="false"><span><br></span></div><div dir="ltr" data-setdir="false"><span>We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can.</span></div><div dir="ltr" data-setdir="false"><span><br></span></div><div dir="ltr" data-setdir="false">With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Here's what things look like normally:</div><div dir="ltr" data-setdir="false"><div><div>Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err<br></div><div>RECVQ-1=2176</div><div>RECVQ-2=0</div><div>RECVQ-3=0</div><div>RECVQ-4=0</div><div>Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv err</div><div>RECVQ-1=0</div><div>RECVQ-2=0</div><div>RECVQ-3=4352</div><div>RECVQ-4=0</div><div>Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv err</div><div>RECVQ-1=2176</div><div>RECVQ-2=0</div><div>RECVQ-3=0</div><div>RECVQ-4=0</div><div><br></div></div>The RECVQs are sparsely used, and there are no errors.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):</div><div>Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv err</div><div>RECVQ-1=1595008</div><div>RECVQ-2=106217088</div><div>RECVQ-3=53694976</div><div>RECVQ-4=31858816</div><div>Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv err</div><div>RECVQ-1=13056</div><div>RECVQ-2=126397312</div><div>RECVQ-3=75568128</div><div>RECVQ-4=41626880</div><div>Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv err</div><div>RECVQ-1=920448</div><div>RECVQ-2=146122752</div><div>RECVQ-3=100951168</div><div>RECVQ-4=52622208</div><div>Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv err</div><div>RECVQ-1=13140864</div><div>RECVQ-2=44494848</div><div>RECVQ-3=125579136</div><div>RECVQ-4=0</div><div><br></div></div><div dir="ltr" data-setdir="false">Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:</div><div><div>Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv err</div><div>RECVQ-1=536871424</div><div>RECVQ-2=200357376</div><div>RECVQ-3=292948352</div><div>RECVQ-4=28890752</div><div>Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv err</div><div>RECVQ-1=536871424</div><div>RECVQ-2=223551360</div><div>RECVQ-3=314995584</div><div>RECVQ-4=41735680</div><div>Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv err</div><div>RECVQ-1=536871424</div><div>RECVQ-2=244732544</div><div>RECVQ-3=338239616</div><div>RECVQ-4=53858176</div><div>Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv err</div><div>RECVQ-1=536871424</div><div>RECVQ-2=265258752</div><div>RECVQ-3=360643712</div><div>RECVQ-4=65362688</div><div><br></div></div>The common denominator I've found is that one of the RECVQs hits <span><span style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (<span><span style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops:</span></span></span></span></div><div dir="ltr" data-setdir="false"><span><span style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;"><span><span style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;"><br></span></span></span></span></div><div dir="ltr" data-setdir="false"><span><span style="color: rgb(0, 0, 0); font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;"><span><div><div>top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32</div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div> 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng</div><div> </div><div>top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31</div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div> 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng</div><div> </div><div>top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31</div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div> 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng</div></div></span></span></span></div><div><br></div><div dir="ltr" data-setdir="false">As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set).</div><div dir="ltr" data-setdir="false"><div><br></div><div dir="ltr" data-setdir="false">Anything else I should try?</div><br></div></div></body></html>