<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Consolas, monospace" size="2">
<div>Also, I hope syslog-ng 3.3 will fix the issue with my syslog-ng.conf having greater than 5000 lines. Syslog-ng 3.0.8 chokes on this (error: memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347). I had to revert back to syslog-ng-2.1.4
which works fine.</div>
<div>...or do I have to still do this which you suggested...</div>
<div> </div>
<div><font face="Courier New, monospace">From: syslog-ng-bounces@lists.balabit.hu [<a href="mailto:syslog-ng-bounces@lists.balabit.hu">mailto:syslog-ng-bounces@lists.balabit.hu</a>] On Behalf Of Balazs Scheidler</font></div>
<div><font face="Courier New, monospace">Sent: Friday, November 26, 2010 6:58 AM</font></div>
<div><font face="Courier New, monospace">To: Syslog-ng users' and developers' mailing list</font></div>
<div><font face="Courier New, monospace">Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347</font></div>
<div><font face="Courier New, monospace"> </font></div>
<div><font face="Courier New, monospace">I've now looked into this. Although the generated parser allocates the</font></div>
<div><font face="Courier New, monospace">stack dynamically, it seems there's a builtin limit of 10k elements by</font></div>
<div><font face="Courier New, monospace">default.</font></div>
<div><font face="Courier New, monospace"> </font></div>
<div><font face="Courier New, monospace">This can be changed by adding the following line to cfg-grammar.y:</font></div>
<div><font face="Courier New, monospace">(somewhere where other defines are listed):</font></div>
<div><font face="Courier New, monospace"> </font></div>
<div><font face="Courier New, monospace">#define YYMAXDEPTH 20000</font></div>
<div><font face="Courier New, monospace"> </font></div>
<div><font face="Courier New, monospace">Assuming that 20k would be enough for your purpose. I really don't see</font></div>
<div><font face="Courier New, monospace">why your configuration file needs so much space, but anyway, this is the</font></div>
<div><font face="Courier New, monospace">way to solve this problem.</font></div>
<div><font face="Courier New, monospace"> </font></div>
<div><font face="Courier New, monospace">After changing .y file you need to recompile (you'll also need bison,</font></div>
<div><font face="Courier New, monospace">since you'd be changing the .y file).</font></div>
<div> </div>
<div>Thanks</div>
<div> </div>
<div># mukarram</div>
<div> </div>
<div>-----Original Message-----<br>
From: syslog-ng-bounces@lists.balabit.hu [<a href="mailto:syslog-ng-bounces@lists.balabit.hu">mailto:syslog-ng-bounces@lists.balabit.hu</a>] On Behalf Of Mukarram Syed<br>
Sent: Monday, December 06, 2010 6:18 AM<br>
To: 'Syslog-ng users' and developers' mailing list'<br>
Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.</div>
<div> </div>
<div>Thank You for the update.</div>
<div> </div>
<div> </div>
<div>-----Original Message-----</div>
<div>From: syslog-ng-bounces@lists.balabit.hu [<a href="mailto:syslog-ng-bounces@lists.balabit.hu">mailto:syslog-ng-bounces@lists.balabit.hu</a>] On Behalf Of Balazs Scheidler</div>
<div>Sent: Saturday, December 04, 2010 5:19 AM</div>
<div>To: Syslog-ng users' and developers' mailing list</div>
<div>Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.</div>
<div> </div>
<div>On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:</div>
<div>> I (and others) got around this problem by launching multiple syslog-ng</div>
<div>> processes and having a master process do very basic filtering and</div>
<div>> routing to the child processes.</div>
<div>> I think I remember hearing that syslog-ng 3.2 was going to use threads</div>
<div>> (at least for destination drivers) but dont quote me on that.</div>
<div> </div>
<div>It is actually syslog-ng 3.3 that's going to use threads. The code which</div>
<div>I've completed so far is a work-in-progress, published at:</div>
<div> </div>
<div><a href="http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll">http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll</a></div>
<div> </div>
<div>Work-in-progress means that I'm rebasing and restructuring the code</div>
<div>quote often.</div>
<div> </div>
<div>Since this was mentioned, I thought I'd mention some numbers how it</div>
<div>goes.</div>
<div> </div>
<div> * I was measuring everything on my development laptop (Core2 Duo,</div>
<div>2.2Ghz, 4GB RAM)</div>
<div> * I was running a full GNOME desktop, running the tests in a</div>
<div>gnome-terminal, I wasn't doing anything, but some jitter can be</div>
<div>associated to background tasks.</div>
<div> * I was running 10 second bursts, with each message containing 256</div>
<div>bytes of data (loggen defaults)</div>
<div> * I was using the tcp() source</div>
<div> * I was playing with various tuning parameters (window-size,</div>
<div>fetch-limit, flush_lines, etc. but I didn't really record their value).</div>
<div> * loggen was running on the same box as syslog-ng itself.</div>
<div> </div>
<div>With these in mind:</div>
<div> </div>
<div>1) single connection, trying to send messages as fast as possible</div>
<div> </div>
<div> syslog-ng 3.2: 58000 msg/sec</div>
<div> syslog-ng 3.3: 180000 msg/sec</div>
<div> </div>
<div>2) also tested how it scales with 1000 clients</div>
<div> </div>
<div> syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were</div>
<div>sending messages</div>
<div> syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were</div>
<div>sending messages (so using epoll instead of GLib's main loop was</div>
<div>definitely worth the effort)</div>
<div> </div>
<div>Of course my development machine has only two cores, and with more cores</div>
<div>and with more clients, it'll probably reach higher msg/sec. I would</div>
<div>estimate that with real server hardware, it'll scale into the 800k-1000k</div>
<div>msg/sec range assuming the disk isn't the bottleneck. (that's</div>
<div>200-250MB/sec, which is not impossible with a proper disk array)</div>
<div> </div>
<div>Anyway, the code isn't ready yet and even crashes (at exit for example),</div>
<div>so definitely not production quality yet :)</div>
<div> </div>
<div>-- </div>
<div>Bazsi</div>
<div> </div>
<div> </div>
<div>______________________________________________________________________________</div>
<div>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a></div>
<div>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng">
http://www.balabit.com/support/documentation/?product=syslog-ng</a></div>
<div>FAQ: <a href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a></div>
<div> </div>
<div>______________________________________________________________________________</div>
<div>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a></div>
<div>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng">
http://www.balabit.com/support/documentation/?product=syslog-ng</a></div>
<div>FAQ: <a href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a></div>
<div> </div>
<div> </div>
</font>
</body>
</html>