[syslog-ng] UDP errors and lost UDP messages

[Matt Zagrabelny]

On Tue, Apr 7, 2015 at 10:48 AM, Evan Rempel <erempel at uvic.ca> wrote:
> Version 3.1 is really old, but my recollection is that since UDP sources do not have a "connection" the fetch limit does not
> have the same meaning. With your time_sleep of 20 milliseconds, , and your fetch limit of 10, you could only
> process 1000ms/20ms * 10 = 500 UDP messages per second.
>
> You would need to remove your time_sleep option, or set your fetch limit much higher.
>
> If I recall correctly, we used a fetch limit of 5000 when running the 3.1 series of syslog-ng.
>
> Since you use a destination template based on the source host name, you could probably use a relatively small
> log_fifo_size because it is a per-destination setting. If you used a 5000 fetch limit, then a log_fifo_size of 500000 would probably be sufficient.
>
> Your log_is_size will be much more critical. I would have to read the manual again to know if this was a per source setting (be careful with connectionless UDP)
> or a global setting, With flow-control enabled, the source is stopped being read, which is fine for TCP sources, but UDP messages still arrive,
> and the OS UDP buffers will start dropping the messages.
>
> The higher releases of syslog-ng 3.5 and 3.6 have *huge* performance gains. If you must stay with 3.1 then
> it might be useful to run two instances. One for TCP sources configured similar to what I have described above, and one
> for UDP sources that do NOT use flow-control. After all, flow control will just make the OS drop the messages anyway.

Hey Evan,

Thanks for the time you took to write up the detailed message. I appreciate it.

I was still hitting issues with the flow-control omitted. But I'll
review the rest of your email and report back.

Cheers!

-m