[syslog-ng] time_sleep() and inadvertent admission control

[Balazs Scheidler]

On Sat, 2008-01-26 at 15:18 +0100, Balazs Scheidler wrote:
> On Thu, 2008-01-24 at 14:18 -0500, John Morrissey wrote:
> > About a year ago, I noticed heavy CPU consumption during certain workloads
> > (many processes sending a small number of log messages per process, such as
> > a busy Postfix machine) due to io_iter spinning very rapidly on poll(). We
> > kludged around it by adding the time_sleep() directive, to add an artificial
> > delay at the end of the io_iter loop and prevent the loop from rolling over
> > too quickly:
> > 
> > http://marc.info/?l=syslog-ng&m=114009552929622&w=2
> > 
> > We started using time_sleep(30) across all of our machines, since that delay
> > value didn't seem to cause any problems for our workloads and we wanted to
> > keep the configuration uniform.
> > 
> > We noticed recently that time_sleep() exhibits some inadvertent admission
> > control behavior. When poll() indicates that the listener socket has
> > activity (new connections), syslog-ng seems to accept() only once on it,
> > allowing one new connection per poll(). As a result, it only allows:
> > 
> >   1000 / time_sleep()
> > 
> > connections per second. Accordingly, with time_sleep(30), only 33
> > connections would be allowed every second. 
> 
> 
> Thanks for the detailed analysis (and for the original idea too :), I
> think the following solutions exist:
> 
>  * do multiple accepts per poll loop; or
>  * increase the I/O priority for the listeners
> 
> The first one easily increases the incoming connection rate and is
> simple to implement, the second is more complex and might cause further
> unexpected behaviour:
> 
> if the priority of the listeners is increased, that would mean that any
> incoming connection might starve the incoming message stream, e.g. if
> there's a continous stream of incoming connections, then long-living
> connections might be starved.
> 
> So I'd choose the first option, what do you think?

Just a quick note to let you know that I've just pushed a patch that
implements multiple accepts per poll iteration.

It's in the 2.1 branch, but it is probably trivial to backport it over
to 2.0, I'm going to do that when a maintenance release for 2.0 is due.

It seems to work for me, and I think it should solve your performance
issues.

http://git.balabit.hu/?p=bazsi/syslog-ng-2.1.git;a=commit;h=33f45d65490c2e8e3c52e468ec733266a0230b13

-- 
Bazsi