[syslog-ng] Multi-core use in threaded mode

[Martin Holste]

Disabling threading definitely improved performance for UDP sources.

On Tue, May 1, 2012 at 6:12 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> Hi,
>
> The syslog-ng team within BalaBit is conducting a detailed performance
> analysis on UDP. I don't have any final results yet though.
>
> When disabling threading for a given udp source, you essentially pin its
> processing to the main thread, essentially removing the latency caused
> by a thread switch between detecting the availability of messages and
> the launch of the worker thread that processes the messages.
>
> This latency can decrease performance, if there's enough time between to
> datagrams for syslog-ng to go to sleep. If incoming datagrams come in
> spikes and syslog-ng is able to run its input processing thread for a
> longer time, then the threaded udp source should be faster.
>
> So disabling threading is not necessarily a win for everyone. I know it
> is not the most reliable behaviour that we have right now, but I have no
> real solutions yet.
>
> On Wed, 2012-04-18 at 10:25 +0200, Sandor Geller wrote:
>> Hi,
>>
>> See the 'Losing to much remote sent logs' thread
>>
>> There are issues with UDP in threaded mode so the workaround is that
>> instead of adding threaded(yes) to the global options it should set at
>> the individual non-UDP sources / destinations.
>>
>> hth,
>>
>> Sandor
>>
>> On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert at balabit.hu> wrote:
>> > The problem with UDP is that it is a connectionless protocol which doesn't scale
>> > to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to
>> > separate the messages to different source or destination drivers, so syslog-ng
>> > will start separate threads for them.
>> >
>> > I don't know if Bazsi is planning to add multiple threads to UDP handling.
>> >
>> > Robert
>> >
>> > On 04/17/2012 09:12 PM, Martin Holste wrote:
>> >
>> >> Yes, threaded(yes) is enabled as confirmed when using top -H to see
>> >> individual threads.  I have one inbound UDP source which gets a
>> >> thread, then one outbound UDP source and a program source.  Each looks
>> >> to get about 20-30% of a CPU, but the sum total of all three can never
>> >> go above 100%.  So, is it individual log {} stanzas that are tied to a
>> >> single CPU?
>> >>
>> >> 2012/4/17 Fekete Róbert<frobert at balabit.hu>:
>> >>> Hi,
>> >>>
>> >>> is threading enabled? It is off by default.
>> >>> Try:
>> >>> options {threaded(yes) ; };
>> >>>
>> >>> Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
>> >>>
>> >>> Regards,
>> >>>
>> >>> Robert
>> >>>
>> >>>
>> >>> On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste at gmail.com>  wrote:
>> >>>
>> >>>> I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises
>> >>>> above 103% and log loss occurs.  Does threaded mode not allow separate
>> >>>> threads to utilize multiple cores?  I need to scale above a single
>> >>>> CPU.
>> >>>> ______________________________________________________________________________
>> >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> >>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> ______________________________________________________________________________
>> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >>>
>> >> ______________________________________________________________________________
>> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >>
>> >>
>> >
>> >
>> > ______________________________________________________________________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>