Multi-core use in threaded mode
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU.
Hi, is threading enabled? It is off by default. Try: options {threaded(yes) ; }; Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source. Regards, Robert On Tuesday, April 17, 2012 20:34 CEST, Martin Holste <mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Yes, threaded(yes) is enabled as confirmed when using top -H to see individual threads. I have one inbound UDP source which gets a thread, then one outbound UDP source and a program source. Each looks to get about 20-30% of a CPU, but the sum total of all three can never go above 100%. So, is it individual log {} stanzas that are tied to a single CPU? 2012/4/17 Fekete Róbert <frobert@balabit.hu>:
Hi,
is threading enabled? It is off by default. Try: options {threaded(yes) ; };
Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
Regards,
Robert
On Tuesday, April 17, 2012 20:34 CEST, Martin Holste <mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The problem with UDP is that it is a connectionless protocol which doesn't scale to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to separate the messages to different source or destination drivers, so syslog-ng will start separate threads for them. I don't know if Bazsi is planning to add multiple threads to UDP handling. Robert On 04/17/2012 09:12 PM, Martin Holste wrote:
Yes, threaded(yes) is enabled as confirmed when using top -H to see individual threads. I have one inbound UDP source which gets a thread, then one outbound UDP source and a program source. Each looks to get about 20-30% of a CPU, but the sum total of all three can never go above 100%. So, is it individual log {} stanzas that are tied to a single CPU?
2012/4/17 Fekete Róbert<frobert@balabit.hu>:
Hi,
is threading enabled? It is off by default. Try: options {threaded(yes) ; };
Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
Regards,
Robert
On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, See the 'Losing to much remote sent logs' thread There are issues with UDP in threaded mode so the workaround is that instead of adding threaded(yes) to the global options it should set at the individual non-UDP sources / destinations. hth, Sandor On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert@balabit.hu> wrote:
The problem with UDP is that it is a connectionless protocol which doesn't scale to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to separate the messages to different source or destination drivers, so syslog-ng will start separate threads for them.
I don't know if Bazsi is planning to add multiple threads to UDP handling.
Robert
On 04/17/2012 09:12 PM, Martin Holste wrote:
Yes, threaded(yes) is enabled as confirmed when using top -H to see individual threads. I have one inbound UDP source which gets a thread, then one outbound UDP source and a program source. Each looks to get about 20-30% of a CPU, but the sum total of all three can never go above 100%. So, is it individual log {} stanzas that are tied to a single CPU?
2012/4/17 Fekete Róbert<frobert@balabit.hu>:
Hi,
is threading enabled? It is off by default. Try: options {threaded(yes) ; };
Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
Regards,
Robert
On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
That explains it, then! Thanks for the clarification. Sounds like I'll have to build some sort of intricate same-host syslog router to multithread the UDP in the meantime, as it's getting very near to 100% of a single CPU at about 20k logs/sec. I don't suppose anyone has a good link to an iptables-based UDP load-balancer? On Wed, Apr 18, 2012 at 3:25 AM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
Hi,
See the 'Losing to much remote sent logs' thread
There are issues with UDP in threaded mode so the workaround is that instead of adding threaded(yes) to the global options it should set at the individual non-UDP sources / destinations.
hth,
Sandor
On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert@balabit.hu> wrote:
The problem with UDP is that it is a connectionless protocol which doesn't scale to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to separate the messages to different source or destination drivers, so syslog-ng will start separate threads for them.
I don't know if Bazsi is planning to add multiple threads to UDP handling.
Robert
On 04/17/2012 09:12 PM, Martin Holste wrote:
Yes, threaded(yes) is enabled as confirmed when using top -H to see individual threads. I have one inbound UDP source which gets a thread, then one outbound UDP source and a program source. Each looks to get about 20-30% of a CPU, but the sum total of all three can never go above 100%. So, is it individual log {} stanzas that are tied to a single CPU?
2012/4/17 Fekete Róbert<frobert@balabit.hu>:
Hi,
is threading enabled? It is off by default. Try: options {threaded(yes) ; };
Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
Regards,
Robert
On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Martin Holste wrote:
That explains it, then! Thanks for the clarification. Sounds like I'll have to build some sort of intricate same-host syslog router to multithread the UDP in the meantime, as it's getting very near to 100% of a single CPU at about 20k logs/sec. I don't suppose anyone has a good link to an iptables-based UDP load-balancer?
Look up the iptables statistics module. It's fairly easily set up. This is a NAT'ing setup I use to alternate traffic between two servers, maybe that'll give you some ideas: iptables -t nat -A OUTPUT -m mark --mark 20 -m statistic --mode nth --every 2 --packet 0 -j DNAT --to $fe1 iptables -t nat -A OUTPUT -m mark --mark 20 -j DNAT --to $fe2 -- Per Jessen, Zürich (6.2°C)
Hi, The syslog-ng team within BalaBit is conducting a detailed performance analysis on UDP. I don't have any final results yet though. When disabling threading for a given udp source, you essentially pin its processing to the main thread, essentially removing the latency caused by a thread switch between detecting the availability of messages and the launch of the worker thread that processes the messages. This latency can decrease performance, if there's enough time between to datagrams for syslog-ng to go to sleep. If incoming datagrams come in spikes and syslog-ng is able to run its input processing thread for a longer time, then the threaded udp source should be faster. So disabling threading is not necessarily a win for everyone. I know it is not the most reliable behaviour that we have right now, but I have no real solutions yet. On Wed, 2012-04-18 at 10:25 +0200, Sandor Geller wrote:
Hi,
See the 'Losing to much remote sent logs' thread
There are issues with UDP in threaded mode so the workaround is that instead of adding threaded(yes) to the global options it should set at the individual non-UDP sources / destinations.
hth,
Sandor
On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert@balabit.hu> wrote:
The problem with UDP is that it is a connectionless protocol which doesn't scale to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to separate the messages to different source or destination drivers, so syslog-ng will start separate threads for them.
I don't know if Bazsi is planning to add multiple threads to UDP handling.
Robert
On 04/17/2012 09:12 PM, Martin Holste wrote:
Yes, threaded(yes) is enabled as confirmed when using top -H to see individual threads. I have one inbound UDP source which gets a thread, then one outbound UDP source and a program source. Each looks to get about 20-30% of a CPU, but the sum total of all three can never go above 100%. So, is it individual log {} stanzas that are tied to a single CPU?
2012/4/17 Fekete Róbert<frobert@balabit.hu>:
Hi,
is threading enabled? It is off by default. Try: options {threaded(yes) ; };
Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
Regards,
Robert
On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Disabling threading definitely improved performance for UDP sources. On Tue, May 1, 2012 at 6:12 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
Hi,
The syslog-ng team within BalaBit is conducting a detailed performance analysis on UDP. I don't have any final results yet though.
When disabling threading for a given udp source, you essentially pin its processing to the main thread, essentially removing the latency caused by a thread switch between detecting the availability of messages and the launch of the worker thread that processes the messages.
This latency can decrease performance, if there's enough time between to datagrams for syslog-ng to go to sleep. If incoming datagrams come in spikes and syslog-ng is able to run its input processing thread for a longer time, then the threaded udp source should be faster.
So disabling threading is not necessarily a win for everyone. I know it is not the most reliable behaviour that we have right now, but I have no real solutions yet.
On Wed, 2012-04-18 at 10:25 +0200, Sandor Geller wrote:
Hi,
See the 'Losing to much remote sent logs' thread
There are issues with UDP in threaded mode so the workaround is that instead of adding threaded(yes) to the global options it should set at the individual non-UDP sources / destinations.
hth,
Sandor
On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert@balabit.hu> wrote:
The problem with UDP is that it is a connectionless protocol which doesn't scale to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to separate the messages to different source or destination drivers, so syslog-ng will start separate threads for them.
I don't know if Bazsi is planning to add multiple threads to UDP handling.
Robert
On 04/17/2012 09:12 PM, Martin Holste wrote:
Yes, threaded(yes) is enabled as confirmed when using top -H to see individual threads. I have one inbound UDP source which gets a thread, then one outbound UDP source and a program source. Each looks to get about 20-30% of a CPU, but the sum total of all three can never go above 100%. So, is it individual log {} stanzas that are tied to a single CPU?
2012/4/17 Fekete Róbert<frobert@balabit.hu>:
Hi,
is threading enabled? It is off by default. Try: options {threaded(yes) ; };
Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
Regards,
Robert
On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste@gmail.com> wrote:
I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (6)
-
Balazs Scheidler
-
Fekete Robert
-
Fekete Róbert
-
Martin Holste
-
Per Jessen
-
Sandor Geller