Hi, If policy does not prevent you from using something outside of EPEL, I'd recommend you to give so-reuseport() a try. It requires a recent syslog-ng. For details check: https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-ove... It's not a silver bullet, but in some situations it might improve the UDP situation considerably. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik On Tue, May 7, 2019 at 3:38 PM Jim Hendrick <james.r.hendrick@gmail.com> wrote:
Good questions. If you are looking for an accurate count, I would go off-box with a tap. Anything that lives on RHEL will in some way be subject to hardware / software limits of the IP stack.
Your config looks reasonable.
I suggest looking into load balancing as a possibility. Maybe an F5 in front of your log servers.
In general, I have seen the same kind of issue, and had unfortunately little success with kernel parameter tuning.
At least UDP fails more politely than TCP :-)
Jim
On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche@gmail.com> wrote:
Hi,
I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes.
we have syslog version 3.5.6.
Questions
- Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make?
Environment:
- RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk
Errors:
Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors
- The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors.
Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq