Hi, I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes. we have syslog version 3.5.6. Questions - Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make? Environment: - RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk Errors: Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors - The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors. Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };
Good questions. If you are looking for an accurate count, I would go off-box with a tap. Anything that lives on RHEL will in some way be subject to hardware / software limits of the IP stack. Your config looks reasonable. I suggest looking into load balancing as a possibility. Maybe an F5 in front of your log servers. In general, I have seen the same kind of issue, and had unfortunately little success with kernel parameter tuning. At least UDP fails more politely than TCP :-) Jim On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche@gmail.com> wrote:
Hi,
I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes.
we have syslog version 3.5.6.
Questions
- Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make?
Environment:
- RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk
Errors:
Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors
- The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors.
Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, If policy does not prevent you from using something outside of EPEL, I'd recommend you to give so-reuseport() a try. It requires a recent syslog-ng. For details check: https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-ove... It's not a silver bullet, but in some situations it might improve the UDP situation considerably. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik On Tue, May 7, 2019 at 3:38 PM Jim Hendrick <james.r.hendrick@gmail.com> wrote:
Good questions. If you are looking for an accurate count, I would go off-box with a tap. Anything that lives on RHEL will in some way be subject to hardware / software limits of the IP stack.
Your config looks reasonable.
I suggest looking into load balancing as a possibility. Maybe an F5 in front of your log servers.
In general, I have seen the same kind of issue, and had unfortunately little success with kernel parameter tuning.
At least UDP fails more politely than TCP :-)
Jim
On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche@gmail.com> wrote:
Hi,
I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes.
we have syslog version 3.5.6.
Questions
- Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make?
Environment:
- RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk
Errors:
Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors
- The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors.
Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Also, I don't see log-fetch-limit() setting in your configuration. It defaults to 10, which means that syslog-ng will go to sleep after 10 processed messages to avoid starving other connections. You can easily increase this value to 100 or even 1000, especially if most of it does is a few connections. udp() uses 1 connection regardless of how many clients are logging to it. On Tue, May 7, 2019 at 9:46 PM Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
If policy does not prevent you from using something outside of EPEL, I'd recommend you to give so-reuseport() a try. It requires a recent syslog-ng. For details check: https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-ove... It's not a silver bullet, but in some situations it might improve the UDP situation considerably.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik
On Tue, May 7, 2019 at 3:38 PM Jim Hendrick <james.r.hendrick@gmail.com> wrote:
Good questions. If you are looking for an accurate count, I would go off-box with a tap. Anything that lives on RHEL will in some way be subject to hardware / software limits of the IP stack.
Your config looks reasonable.
I suggest looking into load balancing as a possibility. Maybe an F5 in front of your log servers.
In general, I have seen the same kind of issue, and had unfortunately little success with kernel parameter tuning.
At least UDP fails more politely than TCP :-)
Jim
On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche@gmail.com> wrote:
Hi,
I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes.
we have syslog version 3.5.6.
Questions
- Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make?
Environment:
- RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk
Errors:
Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors
- The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors.
Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Tue, May 07, 2019 at 11:15:52PM +0800, Swordfish Binary wrote:
- Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)?
FWIW we use collectd, and its protocols plugin provides counters for the following Udp instances: InCsumErrors InDatagrams InErrors NoPorts OutDatagrams RcvbufErrors SndbufErrors If you need anything more, I'd suggest trying out systemtap which should give you more insight, and it's part of RHEL7
participants (5)
-
Czanik, Péter
-
Fabien Wernli
-
Jim Hendrick
-
Scheidler, Balázs
-
Swordfish Binary