Hi, 

I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations.  We have virtual IP's in all the 8 boxes.

we have syslog version 3.5.6.

Questions

• Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on netstat -su output below)?
• Is it possible to receive all packets sent yet still get UDP errors?
• What is a reasonable amount of syslog EPS to expect with this hardware?
• Are there any other tweaks to the Kernel or Syslog-ng to make?

Environment:

• RHEL 7.3
• 16 cores
• 64GB Memory
• 5TB SAS 15k disk

Errors:

Udp:

    1540062710 packets received

    257 packets to unknown port received.

    306945112 packet receive errors

    1378189992 packets sent

    306945112 receive buffer errors

    101 send buffer errors

• The "packets received" is how many were accepted by the listening application and NOT how many were attempted
• The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart)
• The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors.

Tuning Steps

These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now:

We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors

kernel

net.ipv4.udp_rmem_min = 131072

net.ipv4.udp_wmem_min = 131072

net.core.netdev_max_backlog=2000

net.core.rmem_max=67108864

syslog-ng

options {

        sync (5000);

        time_reopen (10);

        time_reap(5);

        long_hostnames (off);

        use_dns (no);

        use_fqdn (no);

        create_dirs (no);

        keep_hostname (yes);

        log_fifo_size (536870912);

        stats_freq(60);

        flush_lines(500);

        flush_timeout(10000);

};

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq