[syslog-ng] Multi-core use in threaded mode

[Martin Holste]

That explains it, then!  Thanks for the clarification.  Sounds like
I'll have to build some sort of intricate same-host syslog router to
multithread the UDP in the meantime, as it's getting very near to 100%
of a single CPU at about 20k logs/sec.  I don't suppose anyone has a
good link to an iptables-based UDP load-balancer?

On Wed, Apr 18, 2012 at 3:25 AM, Sandor Geller
<Sandor.Geller at morganstanley.com> wrote:
> Hi,
>
> See the 'Losing to much remote sent logs' thread
>
> There are issues with UDP in threaded mode so the workaround is that
> instead of adding threaded(yes) to the global options it should set at
> the individual non-UDP sources / destinations.
>
> hth,
>
> Sandor
>
> On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert at balabit.hu> wrote:
>> The problem with UDP is that it is a connectionless protocol which doesn't scale
>> to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to
>> separate the messages to different source or destination drivers, so syslog-ng
>> will start separate threads for them.
>>
>> I don't know if Bazsi is planning to add multiple threads to UDP handling.
>>
>> Robert
>>
>> On 04/17/2012 09:12 PM, Martin Holste wrote:
>>
>>> Yes, threaded(yes) is enabled as confirmed when using top -H to see
>>> individual threads.  I have one inbound UDP source which gets a
>>> thread, then one outbound UDP source and a program source.  Each looks
>>> to get about 20-30% of a CPU, but the sum total of all three can never
>>> go above 100%.  So, is it individual log {} stanzas that are tied to a
>>> single CPU?
>>>
>>> 2012/4/17 Fekete Róbert<frobert at balabit.hu>:
>>>> Hi,
>>>>
>>>> is threading enabled? It is off by default.
>>>> Try:
>>>> options {threaded(yes) ; };
>>>>
>>>> Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
>>>>
>>>> Regards,
>>>>
>>>> Robert
>>>>
>>>>
>>>> On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste at gmail.com>  wrote:
>>>>
>>>>> I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises
>>>>> above 103% and log loss occurs.  Does threaded mode not allow separate
>>>>> threads to utilize multiple cores?  I need to scale above a single
>>>>> CPU.
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>