That explains it, then! Thanks for the clarification. Sounds like I'll have to build some sort of intricate same-host syslog router to multithread the UDP in the meantime, as it's getting very near to 100% of a single CPU at about 20k logs/sec. I don't suppose anyone has a good link to an iptables-based UDP load-balancer? On Wed, Apr 18, 2012 at 3:25 AM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
Hi,
See the 'Losing to much remote sent logs' thread
There are issues with UDP in threaded mode so the workaround is that instead of adding threaded(yes) to the global options it should set at the individual non-UDP sources / destinations.
hth,
Sandor
On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert@balabit.hu> wrote:
The problem with UDP is that it is a connectionless protocol which doesn't scale to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to separate the messages to different source or destination drivers, so syslog-ng will start separate threads for them.
I don't know if Bazsi is planning to add multiple threads to UDP handling.
Robert
On 04/17/2012 09:12 PM, Martin Holste wrote:
Yes, threaded(yes) is enabled as confirmed when using top -H to see individual threads. I have one inbound UDP source which gets a thread, then one outbound UDP source and a program source. Each looks to get about 20-30% of a CPU, but the sum total of all three can never go above 100%. So, is it individual log {} stanzas that are tied to a single CPU?
2012/4/17 Fekete Róbert<frobert@balabit.hu>:
Hi,
is threading enabled? It is off by default. Try: options {threaded(yes) ; };
Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
Regards,
Robert
On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq