On Tue, Apr 7, 2015 at 10:48 AM, Evan Rempel <erempel@uvic.ca> wrote:
Version 3.1 is really old, but my recollection is that since UDP sources do not have a "connection" the fetch limit does not have the same meaning. With your time_sleep of 20 milliseconds, , and your fetch limit of 10, you could only process 1000ms/20ms * 10 = 500 UDP messages per second.
You would need to remove your time_sleep option, or set your fetch limit much higher.
If I recall correctly, we used a fetch limit of 5000 when running the 3.1 series of syslog-ng.
Since you use a destination template based on the source host name, you could probably use a relatively small log_fifo_size because it is a per-destination setting. If you used a 5000 fetch limit, then a log_fifo_size of 500000 would probably be sufficient.
Your log_is_size will be much more critical. I would have to read the manual again to know if this was a per source setting (be careful with connectionless UDP) or a global setting, With flow-control enabled, the source is stopped being read, which is fine for TCP sources, but UDP messages still arrive, and the OS UDP buffers will start dropping the messages.
The higher releases of syslog-ng 3.5 and 3.6 have *huge* performance gains. If you must stay with 3.1 then it might be useful to run two instances. One for TCP sources configured similar to what I have described above, and one for UDP sources that do NOT use flow-control. After all, flow control will just make the OS drop the messages anyway.
Hey Evan, Thanks for the time you took to write up the detailed message. I appreciate it. I was still hitting issues with the flow-control omitted. But I'll review the rest of your email and report back. Cheers! -m