Also, I hope syslog-ng 3.3 will fix the issue with my syslog-ng.conf having greater than 5000 lines. Syslog-ng 3.0.8 chokes on this (error: memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347). I had to revert back to syslog-ng-2.1.4 which works fine. ...or do I have to still do this which you suggested... From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Friday, November 26, 2010 6:58 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 I've now looked into this. Although the generated parser allocates the stack dynamically, it seems there's a builtin limit of 10k elements by default. This can be changed by adding the following line to cfg-grammar.y: (somewhere where other defines are listed): #define YYMAXDEPTH 20000 Assuming that 20k would be enough for your purpose. I really don't see why your configuration file needs so much space, but anyway, this is the way to solve this problem. After changing .y file you need to recompile (you'll also need bison, since you'd be changing the .y file). Thanks # mukarram -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Mukarram Syed Sent: Monday, December 06, 2010 6:18 AM To: 'Syslog-ng users' and developers' mailing list' Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. Thank You for the update. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Saturday, December 04, 2010 5:19 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:
I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that.
It is actually syslog-ng 3.3 that's going to use threads. The code which I've completed so far is a work-in-progress, published at: http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll Work-in-progress means that I'm rebasing and restructuring the code quote often. Since this was mentioned, I thought I'd mention some numbers how it goes. * I was measuring everything on my development laptop (Core2 Duo, 2.2Ghz, 4GB RAM) * I was running a full GNOME desktop, running the tests in a gnome-terminal, I wasn't doing anything, but some jitter can be associated to background tasks. * I was running 10 second bursts, with each message containing 256 bytes of data (loggen defaults) * I was using the tcp() source * I was playing with various tuning parameters (window-size, fetch-limit, flush_lines, etc. but I didn't really record their value). * loggen was running on the same box as syslog-ng itself. With these in mind: 1) single connection, trying to send messages as fast as possible syslog-ng 3.2: 58000 msg/sec syslog-ng 3.3: 180000 msg/sec 2) also tested how it scales with 1000 clients syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were sending messages syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were sending messages (so using epoll instead of GLib's main loop was definitely worth the effort) Of course my development machine has only two cores, and with more cores and with more clients, it'll probably reach higher msg/sec. I would estimate that with real server hardware, it'll scale into the 800k-1000k msg/sec range assuming the disk isn't the bottleneck. (that's 200-250MB/sec, which is not impossible with a proper disk array) Anyway, the code isn't ready yet and even crashes (at exit for example), so definitely not production quality yet :) -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html