[syslog-ng] Multi-core use in threaded mode

[Balazs Scheidler]

Hi,

The syslog-ng team within BalaBit is conducting a detailed performance
analysis on UDP. I don't have any final results yet though.

When disabling threading for a given udp source, you essentially pin its
processing to the main thread, essentially removing the latency caused
by a thread switch between detecting the availability of messages and
the launch of the worker thread that processes the messages.

This latency can decrease performance, if there's enough time between to
datagrams for syslog-ng to go to sleep. If incoming datagrams come in
spikes and syslog-ng is able to run its input processing thread for a
longer time, then the threaded udp source should be faster.

So disabling threading is not necessarily a win for everyone. I know it
is not the most reliable behaviour that we have right now, but I have no
real solutions yet.

On Wed, 2012-04-18 at 10:25 +0200, Sandor Geller wrote:
> Hi,
> 
> See the 'Losing to much remote sent logs' thread
> 
> There are issues with UDP in threaded mode so the workaround is that
> instead of adding threaded(yes) to the global options it should set at
> the individual non-UDP sources / destinations.
> 
> hth,
> 
> Sandor
> 
> On Wed, Apr 18, 2012 at 8:55 AM, Fekete Robert <frobert at balabit.hu> wrote:
> > The problem with UDP is that it is a connectionless protocol which doesn't scale
> > to multiple threads in syslog-ng. If you must stick to UDP, ypu can try to
> > separate the messages to different source or destination drivers, so syslog-ng
> > will start separate threads for them.
> >
> > I don't know if Bazsi is planning to add multiple threads to UDP handling.
> >
> > Robert
> >
> > On 04/17/2012 09:12 PM, Martin Holste wrote:
> >
> >> Yes, threaded(yes) is enabled as confirmed when using top -H to see
> >> individual threads.  I have one inbound UDP source which gets a
> >> thread, then one outbound UDP source and a program source.  Each looks
> >> to get about 20-30% of a CPU, but the sum total of all three can never
> >> go above 100%.  So, is it individual log {} stanzas that are tied to a
> >> single CPU?
> >>
> >> 2012/4/17 Fekete Róbert<frobert at balabit.hu>:
> >>> Hi,
> >>>
> >>> is threading enabled? It is off by default.
> >>> Try:
> >>> options {threaded(yes) ; };
> >>>
> >>> Also, scaling depends on the sources/destinations you use. For example, a tcp source can scale only if you have multiple clients that send messages to the same source.
> >>>
> >>> Regards,
> >>>
> >>> Robert
> >>>
> >>>
> >>> On Tuesday, April 17, 2012 20:34 CEST, Martin Holste<mcholste at gmail.com>  wrote:
> >>>
> >>>> I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises
> >>>> above 103% and log loss occurs.  Does threaded mode not allow separate
> >>>> threads to utilize multiple cores?  I need to scale above a single
> >>>> CPU.
> >>>> ______________________________________________________________________________
> >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> ______________________________________________________________________________
> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>
> >> ______________________________________________________________________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>
> >>
> >
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
> 

-- 
Bazsi