[syslog-ng] time_sleep() and inadvertent admission control

[Balazs Scheidler]

On Mon, 2008-01-28 at 10:26 -0800, Evan Rempel wrote:
> Balazs Scheidler wrote:
> > On Thu, 2008-01-24 at 14:18 -0500, John Morrissey wrote:
> >> About a year ago, I noticed heavy CPU consumption during certain workloads
> >> (many processes sending a small number of log messages per process, such as
> >> a busy Postfix machine) due to io_iter spinning very rapidly on poll(). We
> >> kludged around it by adding the time_sleep() directive, to add an artificial
> >> delay at the end of the io_iter loop and prevent the loop from rolling over
> >> too quickly:
> >>
> >> http://marc.info/?l=syslog-ng&m=114009552929622&w=2
> >>
> >> We started using time_sleep(30) across all of our machines, since that delay
> >> value didn't seem to cause any problems for our workloads and we wanted to
> >> keep the configuration uniform.
> >>
> >> We noticed recently that time_sleep() exhibits some inadvertent admission
> >> control behavior. When poll() indicates that the listener socket has
> >> activity (new connections), syslog-ng seems to accept() only once on it,
> >> allowing one new connection per poll(). As a result, it only allows:
> >>
> >>   1000 / time_sleep()
> >>
> >> connections per second. Accordingly, with time_sleep(30), only 33
> >> connections would be allowed every second. 
> > 
> > 
> > Thanks for the detailed analysis (and for the original idea too :), I
> > think the following solutions exist:
> > 
> >  * do multiple accepts per poll loop; or
> >  * increase the I/O priority for the listeners
> > 
> > The first one easily increases the incoming connection rate and is
> > simple to implement, the second is more complex and might cause further
> > unexpected behaviour:
> > 
> > if the priority of the listeners is increased, that would mean that any
> > incoming connection might starve the incoming message stream, e.g. if
> > there's a continous stream of incoming connections, then long-living
> > connections might be starved.
> > 
> > So I'd choose the first option, what do you think?
> 
> Would it be possible to have an acceptor thread that does not use the time_sleep()
> and let the I/O reader threads honor this setting? That way there would
> not be any starvation and there would not be any interaction from time_sleep() and accepting
> connections.
> 
> Just throwing this idea out there as I am not aware of the architecture of the code.

The same can be achieved by much simpler changes. I could do a poll() call for all 
listener fds and if that returns nothing in 10msecs, I'd sleep for time_sleep()-10msec 
and then poll for everything, but I'm not sure this is worth it. Doing multiple 
accepts in each poll iteration should be just as good.

Syslog-ng is not multithreaded now (this is something I'm planning to change).

-- 
Bazsi