[syslog-ng] Leaking objects in size-4096 cache in slabinfo

[Balazs Scheidler]

On Mon, 2010-01-11 at 11:38 -0500, Doug Warner wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 01/05/2010 11:53 AM, Doug Warner wrote:
> > On 12/21/2009 09:41 AM, Doug Warner wrote:
> >> On 12/21/2009 02:50 AM, Balazs Scheidler wrote:
> >>>> Periodically when syslog-ng seems to be working fine we'll see syslog-ng using
> >>>>> a bit of CPU (~20%), but more commonly right now we see it using very little
> >>>>> and the recv-q building up.
> >>>>>
> >>>>> Non of our src/dest in syslog-ng have any flags set other than the above
> >>>>> global options; so whatever is the default for 3.0.4 OSE should be what we're
> >>>>> using.
> >>> Strange. Can you start strace on the syslog-ng process, preferably with
> >>> timestamps (-ttT -s 256 -p <pid>) options to see what it is doing?
> > 
> 
> Just as a follow-up, I was able to solve this by setting flow-control on my
> log path and increasing my log_fetch_limit() and log_iw_size() on my tcp
> source and increasing log_fifo_size() globally.  Doing these 4 things brought
> my memory consumption back in line and syslog-ng seems to be processing the
> logs at the proper pace again.

Great to know, and sorry for not answering any sooner. I was a bit
overwhelmed with other stuff.

Since the settings that you changed are somewhat contradictory, I guess
a smaller set of changes would have resolved the issue as well.

log_fetch_limit()  - with this you increased the number of messages
                     syslog-ng would receive with a single main loop
                     iteration. the default value here is 10, which is 
                     quite conservative. If you have a lot of 
                     connections, increasing this value to a very 
                     high number could make some connections to starve.

Enabling flow control will only help throughput if otherwise syslog-ng 
drops messages actively (e.g. receive it and then decide there's no 
space in the destination FIFO). If you enable flow control, the kernel
would drop/stall messages (instead of syslog-ng), which makes syslog-ng to
care about output messages instead of receiving them just to drop them.

I think in your specific case the increase of fetch-limit was the solution. 
But increasing  fifo_size() when fetch_limit() is increased is a wise thing 
to do. I doubt enabling flow-control would have a real effect.



-- 
Bazsi