Also, I don't see log-fetch-limit() setting in your configuration. It defaults to 10, which means that syslog-ng will go to sleep after 10 processed messages to avoid starving other connections. You can easily increase this value to 100 or even 1000, especially if most of it does is a few connections. udp() uses 1 connection regardless of how many clients are logging to it.


On Tue, May 7, 2019 at 9:46 PM Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,

If policy does not prevent you from using something outside of EPEL, I'd recommend you to give so-reuseport() a try. It requires a recent syslog-ng. For details check: https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-over-udp It's not a silver bullet, but in some situations it might improve the UDP situation considerably.

Bye,

Peter Czanik (CzP) <peter.czanik@balabit.com>
Balabit (a OneIdentity company) / syslog-ng upstream
https://syslog-ng.com/community/
https://twitter.com/PCzanik


On Tue, May 7, 2019 at 3:38 PM Jim Hendrick <james.r.hendrick@gmail.com> wrote:
Good questions. If you are looking for an accurate count, I would go off-box with a tap. Anything that lives on RHEL will in some way be subject to hardware / software limits of the IP stack.

Your config looks reasonable.

I suggest looking into load balancing as a possibility. Maybe an F5 in front of your log servers.

In general, I have seen the same kind of issue, and had unfortunately little success with kernel parameter tuning.

At least UDP fails more politely than TCP :-)

Jim


On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche@gmail.com> wrote:
Hi, 

I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations.  We have virtual IP's in all the 8 boxes.

we have syslog version 3.5.6.

Questions

  • Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on netstat -su output below)?
  • Is it possible to receive all packets sent yet still get UDP errors?
  • What is a reasonable amount of syslog EPS to expect with this hardware?
  • Are there any other tweaks to the Kernel or Syslog-ng to make?
Environment:

  • RHEL 7.3
  • 16 cores
  • 64GB Memory
  • 5TB SAS 15k disk

Errors:

Udp:
    1540062710 packets received
    257 packets to unknown port received.
    306945112 packet receive errors
    1378189992 packets sent
    306945112 receive buffer errors
    101 send buffer errors

  • The "packets received" is how many were accepted by the listening application and NOT how many were attempted
  • The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart)
  • The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors.
Tuning Steps
These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now:
We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors
kernel
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
net.core.netdev_max_backlog=2000
net.core.rmem_max=67108864
syslog-ng
options {
        sync (5000);
        time_reopen (10);
        time_reap(5);
        long_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
        log_fifo_size (536870912);
        stats_freq(60);
        flush_lines(500);
        flush_timeout(10000);
};


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq