[syslog-ng] UDP alerts in syslog

[Jim Hendrick]

Good questions. If you are looking for an accurate count, I would go
off-box with a tap. Anything that lives on RHEL will in some way be subject
to hardware / software limits of the IP stack.

Your config looks reasonable.

I suggest looking into load balancing as a possibility. Maybe an F5 in
front of your log servers.

In general, I have seen the same kind of issue, and had unfortunately
little success with kernel parameter tuning.

At least UDP fails more politely than TCP :-)

Jim


On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche at gmail.com>
wrote:

> Hi,
>
> I am having syslog running all udp/514 on 8 machines and we receive 40000
> EPS in a given point in time at each of the 8 locations.  We have virtual
> IP's in all the 8 boxes.
>
> we have syslog version 3.5.6.
>
> Questions
>
>
>    - Besides a packet capture, is there anything in RHEL (built-in
>    preferred) that provides a counter of UDP packets attempted and not just
>    those that are received (see comments on *netstat -su* output below)?
>    - Is it possible to receive all packets sent yet still get UDP errors?
>    - What is a reasonable amount of syslog EPS to expect with this
>    hardware?
>    - Are there any other tweaks to the Kernel or Syslog-ng to make?
>
> Environment:
>
>
>    - RHEL 7.3
>    - 16 cores
>    - 64GB Memory
>    - 5TB SAS 15k disk
>
> Errors:
>
> Udp:
>     1540062710 packets received
>     257 packets to unknown port received.
>     306945112 packet receive errors
>     1378189992 packets sent
>     306945112 receive buffer errors
>     101 send buffer errors
>
>
>    - The "packets received" is how many were accepted by the listening
>    application and NOT how many were attempted
>    - The "packets to unknown port" is what came in for the application
>    when the application was down or not listening (e.g. during a restart)
>    - The "packet receive errors" clearly indicates an error; HOWEVER, it
>    is not a one-for-one packets-to-error. You can have many more errors than
>    packets were sent, indicating it is possible for a single packet to
>    generate multiple errors.
>
> Tuning Steps
> These are similar to what was done; however, multiple values were tried so
> the numbers below are not exactly what is in production now:
> We tried turning off the udp/514 forwarding to the other applications but
> we did not see a noticeable drop in errors
> kernel
> net.ipv4.udp_rmem_min = 131072
> net.ipv4.udp_wmem_min = 131072
> net.core.netdev_max_backlog=2000
> net.core.rmem_max=67108864
> syslog-ng
> options {
>         sync (5000);
>         time_reopen (10);
>         time_reap(5);
>         long_hostnames (off);
>         use_dns (no);
>         use_fqdn (no);
>         create_dirs (no);
>         keep_hostname (yes);
>         log_fifo_size (536870912);
>         stats_freq(60);
>         flush_lines(500);
>         flush_timeout(10000);
> };
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190507/4052b38b/attachment-0001.html>