[syslog-ng] Leaking objects in size-4096 cache in slabinfo

[Balazs Scheidler]

On Fri, 2009-12-18 at 09:50 -0500, Doug Warner wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 12/18/2009 03:03 AM, Balazs Scheidler wrote:
> > I've checked the kernel source that sockets and skbuffs have their own
> > slab, thus they wouldn't show up in slab-4096.
> > 
> > However skbuffs data portion is stored in kmalloced memory, thus they
> > would could show up in slab-4096.
> > 
> > The question is much memory is showing in slab-4096 (by mulitplying the
> > counters with 4096 to see how much is actually allocated) and whether
> > they correspond with the receive queues that you see.
> > 
> > Also note that if this is a 32 bit machine with lots of memory (> 4GB),
> > then the additional RAM is actually _reducing_ low memory size, which
> > the slab is using. Can you check /proc/meminfo to see how much low
> > memory you have.
> > 
> 
> We're running x86_64, so all RAM should be low-mem (although we only have 2GB).
> 
> Currently it looks like we have ~326MB in Recv-Q and our size-4096 is ~479MB.

This is reasonable, as the space to be allocated is rounded up, thus it
is possible that syslog-ng is indeed the culprit.

> 
> I guess the big concern here is why our recv-q is so high and what we can do
> to reduce it.
> 
> These are the current global options (nothing set on the src or dest since
> this syslog server only handles the tcp connections):
> 
> options {
>         chain_hostnames(off);
>         dns_cache(yes);
>         log_fifo_size(5000);
>         flush_lines(200);
>         flush_timeout(1000);
>         use_fqdn(yes);
>         stats_freq(300);
> };
> 
> Most destinations are files but we do have one program.  The box doesn't seem
> to be particularly loaded otherwise.
> 
> Do these values seem like we're trying to make syslog-ng cache too much
> information?  Is there other settings we should tweak to try to get the
> messages through syslog-ng faster?

Do you have flow control enabled on any of your log paths? is syslog-ng
burning your CPU, is it CPU bound?

-- 
Bazsi