On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:
I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that.
It is actually syslog-ng 3.3 that's going to use threads. The code which I've completed so far is a work-in-progress, published at: http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll Work-in-progress means that I'm rebasing and restructuring the code quote often. Since this was mentioned, I thought I'd mention some numbers how it goes. * I was measuring everything on my development laptop (Core2 Duo, 2.2Ghz, 4GB RAM) * I was running a full GNOME desktop, running the tests in a gnome-terminal, I wasn't doing anything, but some jitter can be associated to background tasks. * I was running 10 second bursts, with each message containing 256 bytes of data (loggen defaults) * I was using the tcp() source * I was playing with various tuning parameters (window-size, fetch-limit, flush_lines, etc. but I didn't really record their value). * loggen was running on the same box as syslog-ng itself. With these in mind: 1) single connection, trying to send messages as fast as possible syslog-ng 3.2: 58000 msg/sec syslog-ng 3.3: 180000 msg/sec 2) also tested how it scales with 1000 clients syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were sending messages syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were sending messages (so using epoll instead of GLib's main loop was definitely worth the effort) Of course my development machine has only two cores, and with more cores and with more clients, it'll probably reach higher msg/sec. I would estimate that with real server hardware, it'll scale into the 800k-1000k msg/sec range assuming the disk isn't the bottleneck. (that's 200-250MB/sec, which is not impossible with a proper disk array) Anyway, the code isn't ready yet and even crashes (at exit for example), so definitely not production quality yet :) -- Bazsi