[syslog-ng]Syslog-ng 1.6.6 Redhat ES 3.0 - too many open files
syslog-ng@lists.balabit.hu
syslog-ng@lists.balabit.hu
Thu, 7 Apr 2005 10:01:58 -0400
This is a multipart message in MIME format.
--=_alternative 004D18A385256FDC_=
Content-Type: text/plain; charset="US-ASCII"
Any further ideas on fixing/debuggering this?
henry@shoelacecity.com
Sent by: syslog-ng-admin@lists.balabit.hu
03/31/2005 11:56 AM
Please respond to
syslog-ng@lists.balabit.hu
To
syslog-ng@lists.balabit.hu
cc
Subject
Re: [syslog-ng]Syslog-ng 1.6.6 Redhat ES 3.0 - too many open files
I never kill HUP the process. I do a full kill, and then a clean start.
I've been needing to do this daily to prevent the accumulated open sockets
from causing problems. The sockets clear as soon as I kill syslog-ng(as
they should), and start building slowly after a retarting it.
One minor thing I left out is that it is an dual proc machine running the
SMP kernel.
Here's a snippet of the complete log:
Mar 31 10:56:41 scribe01 syslog-ng[17624]: STATS: dropped 14307
Mar 31 10:56:58 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 10:58:01 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 10:59:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:00:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:01:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:02:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:03:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:04:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:05:07 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:06:11 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:06:41 scribe01 syslog-ng[17624]: STATS: dropped 15328
Mar 31 11:07:11 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:08:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:09:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:10:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:11:14 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Mar 31 11:12:15 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation
failed: Too many open files
)
Balazs Scheidler <bazsi@balabit.hu>
Sent by: syslog-ng-admin@lists.balabit.hu
03/30/2005 02:59 AM
Please respond to
syslog-ng@lists.balabit.hu
To
syslog-ng@lists.balabit.hu
cc
Subject
Re: [syslog-ng]Syslog-ng 1.6.6 Redhat ES 3.0 - too many open files
On Wed, 2005-03-30 at 09:51 +0200, Balazs Scheidler wrote:
> On Tue, 2005-03-29 at 13:08 -0500, henry@shoelacecity.com wrote:
> >
> >
> >
> > My syslog-ng.conf specifies only 4 real log file to write to, and one
> > pipe(for mysql writing), and two UDP destinations (spoof enabled).
> >
> > CPU utilization on the machine is less than 5%, and there's plenty of
> > free memory.
>
> Hmmm it is strange, you most certainly have an fd leak, even though the
> libnet context (ie. the raw socket) is initialized at destination
> initialization time and destroyed at deinitialization time, seemingly
> properly.
>
> This is only done on initialization and configuration file reload. Are
> you HUP-ing syslog-ng very often? I can't see how so many raw sockets
> accumulated, assuming that one fd is leaked for each HUP.
>
Hmm.. I've now tried to reproduce the problem but without success, I
created a spoof-source enabled UDP destination, sent a couple of
messages then sent a HUP to syslog-ng, again a couple of messages, HUP
and so on a couple of times.
The end result was that I had a single raw socket opened. I'm still
curious how many times you HUP syslog-ng in one week to have so many raw
sockets accumulated.
Either there's a problem in your libnet library, or something triggers a
reinit within syslog-ng without tearing down the previous instance. But
I can't reproduce that here.
Another related question is that I see this in your logs that you sent:
syslog-ng[15710]: STATS: dropped 12828
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open
files)
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open
files)
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open
files)
Can you add the timestamps for these, just to see the interval between
those?
--
Bazsi
_______________________________________________
syslog-ng maillist - syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
--=_alternative 004D18A385256FDC_=
Content-Type: text/html; charset="US-ASCII"
<br><font size=2 face="sans-serif">Any further ideas on fixing/debuggering
this?</font>
<br>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>henry@shoelacecity.com</b>
</font>
<br><font size=1 face="sans-serif">Sent by: syslog-ng-admin@lists.balabit.hu</font>
<p><font size=1 face="sans-serif">03/31/2005 11:56 AM</font>
<table border>
<tr valign=top>
<td bgcolor=white>
<div align=center><font size=1 face="sans-serif">Please respond to<br>
syslog-ng@lists.balabit.hu</font></div></table>
<br>
<td width=59%>
<table width=100%>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td valign=top><font size=1 face="sans-serif">syslog-ng@lists.balabit.hu</font>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td valign=top>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td valign=top><font size=1 face="sans-serif">Re: [syslog-ng]Syslog-ng
1.6.6 Redhat ES 3.0 - too many open files</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><font size=2 face="sans-serif"><br>
I never kill HUP the process. I do a full kill, and then a clean
start. I've been needing to do this daily to prevent the accumulated
open sockets from causing problems. The sockets clear as soon as
I kill syslog-ng(as they should), and start building slowly after a retarting
it.</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
One minor thing I left out is that it is an dual proc machine running the
SMP kernel. </font><font size=3><br>
<br>
<br>
</font><font size=2 face="sans-serif"><br>
Here's a snippet of the complete log:</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
Mar 31 10:56:41 scribe01 syslog-ng[17624]: STATS: dropped 14307</font><font size=3>
</font><font size=2 face="sans-serif"><br>
Mar 31 10:56:58 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 10:58:01 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 10:59:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:00:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:01:03 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:02:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:03:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:04:06 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:05:07 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:06:11 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:06:41 scribe01 syslog-ng[17624]: STATS: dropped 15328</font><font size=3>
</font><font size=2 face="sans-serif"><br>
Mar 31 11:07:11 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:08:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:09:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:10:13 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:11:14 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> </font><font size=2 face="sans-serif"><br>
Mar 31 11:12:15 scribe01 syslog-ng[17624]: Error initializing raw socket,
spoof-source support disabled. (libnet_open_raw: SOCK_RAW allocation failed:
Too many open files</font><font size=3> </font><font size=2 face="sans-serif"><br>
)</font><font size=3> <br>
<br>
</font>
<table width=100%>
<tr valign=top>
<td width=37%><font size=1 face="sans-serif"><b>Balazs Scheidler <bazsi@balabit.hu></b>
<br>
Sent by: syslog-ng-admin@lists.balabit.hu</font><font size=3> </font>
<p><font size=1 face="sans-serif">03/30/2005 02:59 AM</font><font size=3>
</font>
<br>
<table border=4 width=100%>
<tr valign=top>
<td width=100% bgcolor=white>
<div align=center><font size=1 face="sans-serif">Please respond to<br>
syslog-ng@lists.balabit.hu</font></div></table>
<p>
<td width=62%>
<table width=100%>
<tr>
<td width=13%>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td width=86% valign=top><font size=1 face="sans-serif">syslog-ng@lists.balabit.hu</font><font size=3>
</font>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td valign=top>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td valign=top><font size=1 face="sans-serif">Re: [syslog-ng]Syslog-ng
1.6.6 Redhat ES 3.0 - too many open files</font></table>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=49%>
<td width=50%></table>
<br></table>
<br><font size=3><br>
<br>
</font><font size=2><tt><br>
On Wed, 2005-03-30 at 09:51 +0200, Balazs Scheidler wrote:<br>
> On Tue, 2005-03-29 at 13:08 -0500, henry@shoelacecity.com wrote:<br>
> > <br>
> > <br>
> > <br>
> > My syslog-ng.conf specifies only 4 real log file to write to,
and one<br>
> > pipe(for mysql writing), and two UDP destinations (spoof enabled).
<br>
> > <br>
> > CPU utilization on the machine is less than 5%, and there's plenty
of<br>
> > free memory. <br>
> <br>
> Hmmm it is strange, you most certainly have an fd leak, even though
the<br>
> libnet context (ie. the raw socket) is initialized at destination<br>
> initialization time and destroyed at deinitialization time, seemingly<br>
> properly.<br>
> <br>
> This is only done on initialization and configuration file reload.
Are<br>
> you HUP-ing syslog-ng very often? I can't see how so many raw sockets<br>
> accumulated, assuming that one fd is leaked for each HUP.<br>
> <br>
<br>
Hmm.. I've now tried to reproduce the problem but without success, I<br>
created a spoof-source enabled UDP destination, sent a couple of<br>
messages then sent a HUP to syslog-ng, again a couple of messages, HUP<br>
and so on a couple of times.<br>
<br>
The end result was that I had a single raw socket opened. I'm still<br>
curious how many times you HUP syslog-ng in one week to have so many raw<br>
sockets accumulated.<br>
<br>
Either there's a problem in your libnet library, or something triggers
a<br>
reinit within syslog-ng without tearing down the previous instance. But<br>
I can't reproduce that here.<br>
<br>
Another related question is that I see this in your logs that you sent:<br>
<br>
<br>
syslog-ng[15710]: STATS: dropped 12828 <br>
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open files)
<br>
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open files)
<br>
syslog-ng[15710]: Error initializing raw socket, spoof- source support
disabled. (libnet_open_raw: SOCK_RAW allocation failed: Too many open files)<br>
<br>
Can you add the timestamps for these, just to see the interval between
<br>
those?<br>
<br>
-- <br>
Bazsi<br>
<br>
<br>
_______________________________________________<br>
syslog-ng maillist - syslog-ng@lists.balabit.hu<br>
https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html<br>
</tt></font><font size=3><br>
</font>
<br>
--=_alternative 004D18A385256FDC_=--