[syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.

[Balazs Scheidler]

On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:
> I (and others) got around this problem by launching multiple syslog-ng
> processes and having a master process do very basic filtering and
> routing to the child processes.
> I think I remember hearing that syslog-ng 3.2 was going to use threads
> (at least for destination drivers) but dont quote me on that.

It is actually syslog-ng 3.3 that's going to use threads. The code which
I've completed so far is a work-in-progress, published at:

http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll

Work-in-progress means that I'm rebasing and restructuring the code
quote often.

Since this was mentioned, I thought I'd mention some numbers how it
goes.

 * I was measuring everything on my development laptop (Core2 Duo,
2.2Ghz, 4GB RAM)
 * I was running a full GNOME desktop, running the tests in a
gnome-terminal, I wasn't doing anything, but some jitter can be
associated to background tasks.
  * I was running 10 second bursts, with each message containing 256
bytes of data (loggen defaults)
  * I was using the tcp() source
  * I was playing with various tuning parameters (window-size,
fetch-limit, flush_lines, etc. but I didn't really record their value).
  * loggen was running on the same box as syslog-ng itself.

With these in mind:

1) single connection, trying to send messages as fast as possible

   syslog-ng 3.2:  58000 msg/sec
   syslog-ng 3.3: 180000 msg/sec

2) also tested how it scales with 1000 clients

   syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were
sending messages
   syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were
sending messages (so using epoll instead of GLib's main loop was
definitely worth the effort)

Of course my development machine has only two cores, and with more cores
and with more clients, it'll probably reach higher msg/sec. I would
estimate that with real server hardware, it'll scale into the 800k-1000k
msg/sec range assuming the disk isn't the bottleneck. (that's
200-250MB/sec, which is not impossible with a proper disk array)

Anyway, the code isn't ready yet and even crashes (at exit for example),
so definitely not production quality yet :)

-- 
Bazsi