[syslog-ng] UDP alerts in syslog

[Swordfish Binary]

Hi,

I am having syslog running all udp/514 on 8 machines and we receive 40000
EPS in a given point in time at each of the 8 locations.  We have virtual
IP's in all the 8 boxes.

we have syslog version 3.5.6.

Questions


   - Besides a packet capture, is there anything in RHEL (built-in
   preferred) that provides a counter of UDP packets attempted and not just
   those that are received (see comments on *netstat -su* output below)?
   - Is it possible to receive all packets sent yet still get UDP errors?
   - What is a reasonable amount of syslog EPS to expect with this hardware?
   - Are there any other tweaks to the Kernel or Syslog-ng to make?

Environment:


   - RHEL 7.3
   - 16 cores
   - 64GB Memory
   - 5TB SAS 15k disk

Errors:

Udp:
    1540062710 packets received
    257 packets to unknown port received.
    306945112 packet receive errors
    1378189992 packets sent
    306945112 receive buffer errors
    101 send buffer errors


   - The "packets received" is how many were accepted by the listening
   application and NOT how many were attempted
   - The "packets to unknown port" is what came in for the application when
   the application was down or not listening (e.g. during a restart)
   - The "packet receive errors" clearly indicates an error; HOWEVER, it is
   not a one-for-one packets-to-error. You can have many more errors than
   packets were sent, indicating it is possible for a single packet to
   generate multiple errors.

Tuning Steps
These are similar to what was done; however, multiple values were tried so
the numbers below are not exactly what is in production now:
We tried turning off the udp/514 forwarding to the other applications but
we did not see a noticeable drop in errors
kernel
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
net.core.netdev_max_backlog=2000
net.core.rmem_max=67108864
syslog-ng
options {
        sync (5000);
        time_reopen (10);
        time_reap(5);
        long_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
        log_fifo_size (536870912);
        stats_freq(60);
        flush_lines(500);
        flush_timeout(10000);
};
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190507/e19dae38/attachment.html>