On Wed, 2009-12-16 at 09:54 -0500, Doug Warner wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/15/2009 03:17 PM, Balazs Scheidler wrote:
Possibly a socket structure (but AFAIR it has its own slab). Can you check if: * syslog-ng has too many open sockets (visible for example with lsof)
We have about 290 open TCP connections.
* whether it reads its input queue (although network packets would again not show up in the slab), using netstat -antp
Most have smallish Recv-Q's (<2000KB), but a couple hosts have receive queues of 50KB, and one of >1MB. Is this an indication that my syslog-ng instance is overloaded and can't get the data out to disk fast enough?
* possibly check the network statistics: netstat -ns
Nothing looks too odd here.
Thanks for your help debugging this issue!
I've checked the kernel source that sockets and skbuffs have their own slab, thus they wouldn't show up in slab-4096. However skbuffs data portion is stored in kmalloced memory, thus they would could show up in slab-4096. The question is much memory is showing in slab-4096 (by mulitplying the counters with 4096 to see how much is actually allocated) and whether they correspond with the receive queues that you see. Also note that if this is a 32 bit machine with lots of memory (> 4GB), then the additional RAM is actually _reducing_ low memory size, which the slab is using. Can you check /proc/meminfo to see how much low memory you have. -- Bazsi