On Fri, 2009-12-18 at 09:50 -0500, Doug Warner wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/18/2009 03:03 AM, Balazs Scheidler wrote:
I've checked the kernel source that sockets and skbuffs have their own slab, thus they wouldn't show up in slab-4096.
However skbuffs data portion is stored in kmalloced memory, thus they would could show up in slab-4096.
The question is much memory is showing in slab-4096 (by mulitplying the counters with 4096 to see how much is actually allocated) and whether they correspond with the receive queues that you see.
Also note that if this is a 32 bit machine with lots of memory (> 4GB), then the additional RAM is actually _reducing_ low memory size, which the slab is using. Can you check /proc/meminfo to see how much low memory you have.
We're running x86_64, so all RAM should be low-mem (although we only have 2GB).
Currently it looks like we have ~326MB in Recv-Q and our size-4096 is ~479MB.
This is reasonable, as the space to be allocated is rounded up, thus it is possible that syslog-ng is indeed the culprit.
I guess the big concern here is why our recv-q is so high and what we can do to reduce it.
These are the current global options (nothing set on the src or dest since this syslog server only handles the tcp connections):
options { chain_hostnames(off); dns_cache(yes); log_fifo_size(5000); flush_lines(200); flush_timeout(1000); use_fqdn(yes); stats_freq(300); };
Most destinations are files but we do have one program. The box doesn't seem to be particularly loaded otherwise.
Do these values seem like we're trying to make syslog-ng cache too much information? Is there other settings we should tweak to try to get the messages through syslog-ng faster?
Do you have flow control enabled on any of your log paths? is syslog-ng burning your CPU, is it CPU bound? -- Bazsi