henry@shoelacecity.com Sent by: syslog-ng-admin@lists.balabit.hu
03/31/2005 11:56 AM
Please respond to
syslog-ng@lists.balabit.hu
To
syslog-ng@lists.balabit.hu
cc
Subject
Re: [syslog-ng]Syslog-ng
1.6.6 Redhat ES 3.0 - too many open files
I never kill HUP the process. I do a full kill, and then a clean
start. I've been needing to do this daily to prevent the accumulated
open sockets from causing problems. The sockets clear as soon as
I kill syslog-ng(as they should), and start building slowly after a retarting
it.
One minor thing I left out is that it is an dual proc machine running the
SMP kernel.
Here's a snippet of the complete log:
Mar 31 10:56:41 scribe01 syslog-ng[17624]: STATS: dropped 14307
Mar 31 10:56:58 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 10:58:01 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 10:59:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:00:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:01:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:02:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:03:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:04:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:05:07 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:06:11 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:06:41 scribe01 syslog-ng[17624]: STATS: dropped 15328
Mar 31 11:07:11 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:08:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:09:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:10:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:11:14 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Mar 31 11:12:15 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files
)
Balazs Scheidler <bazsi@balabit.hu>
Sent by: syslog-ng-admin@lists.balabit.hu
03/30/2005 02:59 AM
Please respond to
syslog-ng@lists.balabit.hu
To
syslog-ng@lists.balabit.hu
cc
Subject
Re: [syslog-ng]Syslog-ng
1.6.6 Redhat ES 3.0 - too many open files
On Wed, 2005-03-30 at 09:51 +0200, Balazs Scheidler wrote:
> On Tue, 2005-03-29 at 13:08 -0500, henry@shoelacecity.com wrote:
> >
> >
> >
> > My syslog-ng.conf specifies only 4 real log file to write to,
and one
> > pipe(for mysql writing), and two UDP destinations (spoof enabled).
> >
> > CPU utilization on the machine is less than 5%, and there's plenty
of
> > free memory.
>
> Hmmm it is strange, you most certainly have an fd leak, even though
the
> libnet context (ie. the raw socket) is initialized at destination
> initialization time and destroyed at deinitialization time, seemingly
> properly.
>
> This is only done on initialization and configuration file reload.
Are
> you HUP-ing syslog-ng very often? I can't see how so many raw sockets
> accumulated, assuming that one fd is leaked for each HUP.
>
Hmm.. I've now tried to reproduce the problem but without success, I
created a spoof-source enabled UDP destination, sent a couple of
messages then sent a HUP to syslog-ng, again a couple of messages, HUP
and so on a couple of times.
The end result was that I had a single raw socket opened. I'm still
curious how many times you HUP syslog-ng in one week to have so many raw
sockets accumulated.
Either there's a problem in your libnet library, or something triggers
a
reinit within syslog-ng without tearing down the previous instance. But
I can't reproduce that here.
Another related question is that I see this in your logs that you sent:
syslog-ng[15710]: STATS: dropped 12828
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open files)
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open files)
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open files)
Can you add the timestamps for these, just to see the interval between
those?