[syslog-ng] UDP alerts in syslog

[Czanik, Péter]

Hi,

If policy does not prevent you from using something outside of EPEL, I'd
recommend you to give so-reuseport() a try. It requires a recent syslog-ng.
For details check:
https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-over-udp
It's not a silver bullet, but in some situations it might improve the UDP
situation considerably.

Bye,

Peter Czanik (CzP) <peter.czanik at balabit.com>
Balabit (a OneIdentity company) / syslog-ng upstream
https://syslog-ng.com/community/
https://twitter.com/PCzanik


On Tue, May 7, 2019 at 3:38 PM Jim Hendrick <james.r.hendrick at gmail.com>
wrote:

> Good questions. If you are looking for an accurate count, I would go
> off-box with a tap. Anything that lives on RHEL will in some way be subject
> to hardware / software limits of the IP stack.
>
> Your config looks reasonable.
>
> I suggest looking into load balancing as a possibility. Maybe an F5 in
> front of your log servers.
>
> In general, I have seen the same kind of issue, and had unfortunately
> little success with kernel parameter tuning.
>
> At least UDP fails more politely than TCP :-)
>
> Jim
>
>
> On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche at gmail.com>
> wrote:
>
>> Hi,
>>
>> I am having syslog running all udp/514 on 8 machines and we receive
>> 40000 EPS in a given point in time at each of the 8 locations.  We have
>> virtual IP's in all the 8 boxes.
>>
>> we have syslog version 3.5.6.
>>
>> Questions
>>
>>
>>    - Besides a packet capture, is there anything in RHEL (built-in
>>    preferred) that provides a counter of UDP packets attempted and not just
>>    those that are received (see comments on *netstat -su* output below)?
>>    - Is it possible to receive all packets sent yet still get UDP errors?
>>    - What is a reasonable amount of syslog EPS to expect with this
>>    hardware?
>>    - Are there any other tweaks to the Kernel or Syslog-ng to make?
>>
>> Environment:
>>
>>
>>    - RHEL 7.3
>>    - 16 cores
>>    - 64GB Memory
>>    - 5TB SAS 15k disk
>>
>> Errors:
>>
>> Udp:
>>     1540062710 packets received
>>     257 packets to unknown port received.
>>     306945112 packet receive errors
>>     1378189992 packets sent
>>     306945112 receive buffer errors
>>     101 send buffer errors
>>
>>
>>    - The "packets received" is how many were accepted by the listening
>>    application and NOT how many were attempted
>>    - The "packets to unknown port" is what came in for the application
>>    when the application was down or not listening (e.g. during a restart)
>>    - The "packet receive errors" clearly indicates an error; HOWEVER, it
>>    is not a one-for-one packets-to-error. You can have many more errors than
>>    packets were sent, indicating it is possible for a single packet to
>>    generate multiple errors.
>>
>> Tuning Steps
>> These are similar to what was done; however, multiple values were tried
>> so the numbers below are not exactly what is in production now:
>> We tried turning off the udp/514 forwarding to the other applications but
>> we did not see a noticeable drop in errors
>> kernel
>> net.ipv4.udp_rmem_min = 131072
>> net.ipv4.udp_wmem_min = 131072
>> net.core.netdev_max_backlog=2000
>> net.core.rmem_max=67108864
>> syslog-ng
>> options {
>>         sync (5000);
>>         time_reopen (10);
>>         time_reap(5);
>>         long_hostnames (off);
>>         use_dns (no);
>>         use_fqdn (no);
>>         create_dirs (no);
>>         keep_hostname (yes);
>>         log_fifo_size (536870912);
>>         stats_freq(60);
>>         flush_lines(500);
>>         flush_timeout(10000);
>> };
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190507/e643f63b/attachment.html>