[syslog-ng] time_sleep() and inadvertent admission control

[Balazs Scheidler]

On Thu, 2008-01-24 at 14:18 -0500, John Morrissey wrote:
> About a year ago, I noticed heavy CPU consumption during certain workloads
> (many processes sending a small number of log messages per process, such as
> a busy Postfix machine) due to io_iter spinning very rapidly on poll(). We
> kludged around it by adding the time_sleep() directive, to add an artificial
> delay at the end of the io_iter loop and prevent the loop from rolling over
> too quickly:
> 
> http://marc.info/?l=syslog-ng&m=114009552929622&w=2
> 
> We started using time_sleep(30) across all of our machines, since that delay
> value didn't seem to cause any problems for our workloads and we wanted to
> keep the configuration uniform.
> 
> We noticed recently that time_sleep() exhibits some inadvertent admission
> control behavior. When poll() indicates that the listener socket has
> activity (new connections), syslog-ng seems to accept() only once on it,
> allowing one new connection per poll(). As a result, it only allows:
> 
>   1000 / time_sleep()
> 
> connections per second. Accordingly, with time_sleep(30), only 33
> connections would be allowed every second. 


Thanks for the detailed analysis (and for the original idea too :), I
think the following solutions exist:

 * do multiple accepts per poll loop; or
 * increase the I/O priority for the listeners

The first one easily increases the incoming connection rate and is
simple to implement, the second is more complex and might cause further
unexpected behaviour:

if the priority of the listeners is increased, that would mean that any
incoming connection might starve the incoming message stream, e.g. if
there's a continous stream of incoming connections, then long-living
connections might be starved.

So I'd choose the first option, what do you think?


-- 
Bazsi