[syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.

[Mukarram Syed]

Also, I hope syslog-ng 3.3 will fix the issue with my syslog-ng.conf having greater than 5000 lines.  Syslog-ng 3.0.8 chokes on this (error:  memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347).  I had to revert back to syslog-ng-2.1.4 which works fine.
...or do I have to still do this which you suggested...

From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Friday, November 26, 2010 6:58 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347

I've now looked into this. Although the generated parser allocates the
stack dynamically, it seems there's a builtin limit of 10k elements by
default.

This can be changed by adding the following line to cfg-grammar.y:
(somewhere where other defines are listed):

#define YYMAXDEPTH 20000

Assuming that 20k would be enough for your purpose. I really don't see
why your configuration file needs so much space, but anyway, this is the
way to solve this problem.

After changing .y file you need to recompile (you'll also need bison,
since you'd be changing the .y file).

Thanks

# mukarram

-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Mukarram Syed
Sent: Monday, December 06, 2010 6:18 AM
To: 'Syslog-ng users' and developers' mailing list'
Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.

Thank You for the update.


-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Saturday, December 04, 2010 5:19 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.

On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:
> I (and others) got around this problem by launching multiple syslog-ng
> processes and having a master process do very basic filtering and
> routing to the child processes.
> I think I remember hearing that syslog-ng 3.2 was going to use threads
> (at least for destination drivers) but dont quote me on that.

It is actually syslog-ng 3.3 that's going to use threads. The code which
I've completed so far is a work-in-progress, published at:

http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll

Work-in-progress means that I'm rebasing and restructuring the code
quote often.

Since this was mentioned, I thought I'd mention some numbers how it
goes.

 * I was measuring everything on my development laptop (Core2 Duo,
2.2Ghz, 4GB RAM)
 * I was running a full GNOME desktop, running the tests in a
gnome-terminal, I wasn't doing anything, but some jitter can be
associated to background tasks.
  * I was running 10 second bursts, with each message containing 256
bytes of data (loggen defaults)
  * I was using the tcp() source
  * I was playing with various tuning parameters (window-size,
fetch-limit, flush_lines, etc. but I didn't really record their value).
  * loggen was running on the same box as syslog-ng itself.

With these in mind:

1) single connection, trying to send messages as fast as possible

   syslog-ng 3.2:  58000 msg/sec
   syslog-ng 3.3: 180000 msg/sec

2) also tested how it scales with 1000 clients

   syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were
sending messages
   syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were
sending messages (so using epoll instead of GLib's main loop was
definitely worth the effort)

Of course my development machine has only two cores, and with more cores
and with more clients, it'll probably reach higher msg/sec. I would
estimate that with real server hardware, it'll scale into the 800k-1000k
msg/sec range assuming the disk isn't the bottleneck. (that's
200-250MB/sec, which is not impossible with a proper disk array)

Anyway, the code isn't ready yet and even crashes (at exit for example),
so definitely not production quality yet :)

--
Bazsi


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20101207/b4653597/attachment.htm