[syslog-ng] syslog-ng stops accepting new connections every 100-110 minutes

[Matt Pinkham]

I haven't seen the max-connections message but the ESTABLISHED connections
(from the same source) keeps incrementing every couple of minutes on the
target (even though the sender only ever shows one connection).  The only
other point I had forgotten to mention (and it shouldn't matter) is that
this traffic runs through a Radware (formerly Nortel) Application Switch
2424 (I previously had a similar syslog config but different data stream
running an Alteon 180e with no issues).  The IP 10.10.10.41 is the load
balance IP (VIP).

I upgraded both source and target to 3.0.3 in case that would help (it
hasn't).

SENDER (10.10.10.227)
(syslog-ng.conf snippet)
options {
          time_reopen (2);
          log_fifo_size (10000);
          long_hostnames (off);
          use_dns (no);
          use_fqdn (no);
          create_dirs (yes);
          dir_perm (0755);
          perm (0644);
          chain_hostnames (no);
          keep_hostname (yes);
          stats_freq (3600);
          log_msg_size (65535);
          log_fifo_size (65536);
        };

destination d_data { tcp("10.10.10.41" so_sndbuf(2094752)
so_keepalive(yes)); };

(netstat)
tcp        0      0 10.10.10.227:38370         10.10.10.41:514
ESTABLISHED 2067/syslog-ng


RECEIVER (10.10.10.31)
(syslog-ng.conf snippet)
source remote {
        udp(ip(0.0.0.0) port(514) so_rcvbuf(1048576));
        tcp(ip(0.0.0.0) port(514) max-connections(500) so_rcvbuf(1048576)
so_keepalive(yes));
};

(netstat)
tcp        0      0 0.0.0.0:514                 0.0.0.0:*
LISTEN      2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9501
ESTABLISHED 2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9503
ESTABLISHED 2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9499
ESTABLISHED 2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9509
ESTABLISHED 2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9511
ESTABLISHED 2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9505
ESTABLISHED 2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9507
ESTABLISHED 2086/syslog-ng
tcp        0      0 10.10.10.31:514            10.10.10.227:9513
ESTABLISHED 2086/syslog-ng


On Thu, Jul 30, 2009 at 3:25 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:

> On Wed, 2009-07-29 at 11:22 -0400, Matt Pinkham wrote:
> > For the last 24 hours on versions 2.0.4, 2.1.4, & 3.0.3 syslog-ng will
> > stop taking new connections via a listening port every 100-110 minutes
> > (aka it will hang up immediately).  It will never recover on its own
> > and has to be restarted.  I haven't figured out the exact interval but
> > hopefully that will be close enough to work with (note that the
> > traffic is fairly low -- 10 mesgs/sec -- 500K-600K data/min).  I had a
> > program logging data locally via /dev/log into a named directory and
> > then moved this program to a remote server.  That remote server does
> > not seem to be having an issue.  I have observed this issue on two
> > separate servers (RHEL4.Coolthat were taking this data feed.  I have
> > tried with flush_lines/sync & time_reopen commented out with no
> > difference as well as log_fifo_size, log_mesg_size,so_recvbuf
> > commented out.  There are no obvious messages about why syslog-ng
> > stops working (even with debug and verbose enabled).  Note that these
> > two servers (that stop working) are behind an Alteon 2424 switch
> > (although I have other feeds to other servers working fine behind this
> > switch).  Ideas?  Need more data?
>
>
> does this mean that syslog-ng is closing the connection immediately? I
> see only one reason that causes this: max_connections() limit is
> reached.
>
> try increasing max-connections()
>
> Although this case is logged in syslog-ng's log.
>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>


-- 
Some men see things as they are and ask why. I see things that never were
and ask for initiative rolls.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090730/183982ff/attachment.htm