[syslog-ng] Profiling syslog-ng

[Balazs Scheidler]

On Mon, 2006-02-13 at 10:57 -0500, John Morrissey wrote:
> On Sun, Feb 12, 2006 at 08:07:14PM +0100, Balazs Scheidler wrote:
> On Sun, Feb 12, 2006 at 02:53:55PM +0100, Balazs Scheidler wrote:
> > Strange, it's been a while since I last profiled syslog-ng. What were
> > your build options?
> 
> They're pretty stock. We used the Debian package for 1.6.9, adding 'debug'
> to DEB_BUILD_OPTIONS, which seems to use:
> 
> ./configure --prefix=/ --mandir=/usr/share/man --sysconfdir=/etc \
> 	--enable-debug
> 
> and '-g -Wall' for CFLAGS. We added -pg manually.

It should be ok. Maybe some kind of kernel patch (grsec or something
like that?) 


> > io_iter basically traverses all registered fds and constructs a "struct
> > pollfd" array every time it is invoked. If there are a lot of fds, this
> > might be a problem.
> 
> I agree; poll(2) is fairly cheap(?), so regenerating the struct pollfd is a
> likely suspect. Could the struct pollfd be cached, regenerated every n msec?
> That way latency for new processes isn't too bad, but it keeps syslog-ng
> from spinning so much?
> 
> Or perhaps it could keep a struct pollfd that is only regenerated when file
> descriptors are opened/closed. Our processes are fairly long-lived, so this
> too could keep the struct from being regenerated so often. Even if logging
> processes are not long-lived, I doubt it would be regenerated more often
> than once every second or few.

the problem is that it is not so simple, this code is in libol, it is
generic and the program relies on this behaviour at a lot of places. (to
request a read callback, it simply changes the value for a want_read
value, no functions called to update the pollfd state)

io_iter currently iterates through the registered fds 3 times, first to
call prepare callbacks, then to count the final set of pollfd structs
and then to fill that up. We might spare one of the loops (the counting
part with some reallocs), but the basic structure has to remain the
same.

I have now looked at the development branch of syslog-ng, and the
mainloop in GLib basically does the same, it traverses the registered
sources multiple times for each iteration. (I have not looked that
closely though, it probably spares the counting part)

As it seems syslog-ng does not scale to many parallel connections and
prefers load patterns with less connections and more traffic on those.

I also looked at epoll now, but it is not a trivial change to convert
either libol nor glib to use that. (I also looked at ivykis as well but
again it would be a big change)

I'll see into removing one of the loops which could help a bit.

-- 
Bazsi