[syslog-ng] time_sleep() and inadvertent admission control

[Alexander Clouter]

Hi,

Balazs Scheidler <bazsi at balabit.hu> [20080128 20:52:36 +0100]:
>
> On Mon, 2008-01-28 at 10:26 -0800, Evan Rempel wrote:
> > >
> > > > [snipped]
> > >
> > > Thanks for the detailed analysis (and for the original idea too :), I
> > > think the following solutions exist:
> > > 
> > >  * do multiple accepts per poll loop; or
> > >  * increase the I/O priority for the listeners
> > > 
> > > The first one easily increases the incoming connection rate and is 
> > > simple to implement, the second is more complex and might cause further 
> > > unexpected behaviour:
> > > 
> > > if the priority of the listeners is increased, that would mean that any 
> > > incoming connection might starve the incoming message stream, e.g. if 
> > > there's a continous stream of incoming connections, then long-living 
> > > connections might be starved.
> > > 
> > > So I'd choose the first option, what do you think?
> > 
> > Would it be possible to have an acceptor thread that does not use the 
> > time_sleep() and let the I/O reader threads honor this setting? That way 
> > there would not be any starvation and there would not be any interaction 
> > from time_sleep() and accepting connections.
> > 
> > Just throwing this idea out there as I am not aware of the architecture 
> > of the code.
> 
> The same can be achieved by much simpler changes. I could do a poll() call 
> for all listener fds and if that returns nothing in 10msecs, I'd sleep for 
> time_sleep()-10msec and then poll for everything, but I'm not sure this is 
> worth it. Doing multiple accepts in each poll iteration should be just as 
> good.
> 
Unsure how portable it is, might only be Linux 2.4+ but looks like BSD and 
Solaris might also do this, however the following looks good:

http://www.kegel.com/c10k.html#nb.sigio

Spares you the poll() during idle periods.  Instead you use poll() *only* 
when you know there is something waiting for you.  Of course multiple accepts 
needed but then no need to faff with timers.

For those not aware C10k, its a good read.

> Syslog-ng is not multithreaded now (this is something I'm planning to change).
> 
Is this to make use of now common SMP setups or to help on IO throughput?  If 
it's on the latter I think I read somewhere mmap'ing a file between a single 
writer (network input, one per input fd) and a pool of writers (disk output, 
one per file) would give you shiny performance.  Of course this is a 
completely different architecture than what syslog-ng gives...syslog-ng-ng 
anyone? :)

Cheers

Alex