[syslog-ng] UDP alerts in syslog

Wed May 8 14:57:53 UTC 2019

Also, I don't see log-fetch-limit() setting in your configuration. It
defaults to 10, which means that syslog-ng will go to sleep after 10
processed messages to avoid starving other connections. You can easily
increase this value to 100 or even 1000, especially if most of it does is a
few connections. udp() uses 1 connection regardless of how many clients are
logging to it.

On Tue, May 7, 2019 at 9:46 PM Czanik, Péter <peter.czanik at balabit.com>
wrote:

> Hi,
>
> If policy does not prevent you from using something outside of EPEL, I'd
> recommend you to give so-reuseport() a try. It requires a recent syslog-ng.
> For details check:
> https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-over-udp
> It's not a silver bullet, but in some situations it might improve the UDP
> situation considerably.
>
> Bye,
>
> Peter Czanik (CzP) <peter.czanik at balabit.com>
> Balabit (a OneIdentity company) / syslog-ng upstream
> https://syslog-ng.com/community/
> https://twitter.com/PCzanik
>
>
> On Tue, May 7, 2019 at 3:38 PM Jim Hendrick <james.r.hendrick at gmail.com>
> wrote:
>
>> Good questions. If you are looking for an accurate count, I would go
>> off-box with a tap. Anything that lives on RHEL will in some way be subject
>> to hardware / software limits of the IP stack.
>>
>> Your config looks reasonable.
>>
>> I suggest looking into load balancing as a possibility. Maybe an F5 in
>> front of your log servers.
>>
>> In general, I have seen the same kind of issue, and had unfortunately
>> little success with kernel parameter tuning.
>>
>> At least UDP fails more politely than TCP :-)
>>
>> Jim
>>
>>
>> On Tue, May 7, 2019 at 11:16 AM Swordfish Binary <linuxteche at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am having syslog running all udp/514 on 8 machines and we receive
>>> 40000 EPS in a given point in time at each of the 8 locations.  We have
>>> virtual IP's in all the 8 boxes.
>>>
>>> we have syslog version 3.5.6.
>>>
>>> Questions
>>>
>>>
>>>    - Besides a packet capture, is there anything in RHEL (built-in
>>>    preferred) that provides a counter of UDP packets attempted and not just
>>>    those that are received (see comments on *netstat -su* output below)?
>>>    - Is it possible to receive all packets sent yet still get UDP
>>>    errors?
>>>    - What is a reasonable amount of syslog EPS to expect with this
>>>    hardware?
>>>    - Are there any other tweaks to the Kernel or Syslog-ng to make?
>>>
>>> Environment:
>>>
>>>
>>>    - RHEL 7.3
>>>    - 16 cores
>>>    - 64GB Memory
>>>    - 5TB SAS 15k disk
>>>
>>> Errors:
>>>
>>> Udp:
>>>     1540062710 packets received
>>>     257 packets to unknown port received.
>>>     306945112 packet receive errors
>>>     1378189992 packets sent
>>>     306945112 receive buffer errors
>>>     101 send buffer errors
>>>
>>>
>>>    - The "packets received" is how many were accepted by the listening
>>>    application and NOT how many were attempted
>>>    - The "packets to unknown port" is what came in for the application
>>>    when the application was down or not listening (e.g. during a restart)
>>>    - The "packet receive errors" clearly indicates an error; HOWEVER,
>>>    it is not a one-for-one packets-to-error. You can have many more errors
>>>    than packets were sent, indicating it is possible for a single packet to
>>>    generate multiple errors.
>>>
>>> Tuning Steps
>>> These are similar to what was done; however, multiple values were tried
>>> so the numbers below are not exactly what is in production now:
>>> We tried turning off the udp/514 forwarding to the other applications
>>> but we did not see a noticeable drop in errors
>>> kernel
>>> net.ipv4.udp_rmem_min = 131072
>>> net.ipv4.udp_wmem_min = 131072
>>> net.core.netdev_max_backlog=2000
>>> net.core.rmem_max=67108864
>>> syslog-ng
>>> options {
>>>         sync (5000);
>>>         time_reopen (10);
>>>         time_reap(5);
>>>         long_hostnames (off);
>>>         use_dns (no);
>>>         use_fqdn (no);
>>>         create_dirs (no);
>>>         keep_hostname (yes);
>>>         log_fifo_size (536870912);
>>>         stats_freq(60);
>>>         flush_lines(500);
>>>         flush_timeout(10000);
>>> };
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190508/0fa14efd/attachment-0001.html>