[syslog-ng] syslog-ng stops accepting new connections every 100-110 minutes

[Balazs Scheidler]

On Thu, 2009-07-30 at 09:52 -0400, Matt Pinkham wrote:
> I haven't seen the max-connections message but the ESTABLISHED
> connections (from the same source) keeps incrementing every couple of
> minutes on the target (even though the sender only ever shows one
> connection).  The only other point I had forgotten to mention (and it
> shouldn't matter) is that this traffic runs through a Radware
> (formerly Nortel) Application Switch 2424 (I previously had a similar
> syslog config but different data stream running an Alteon 180e with no
> issues).  The IP 10.10.10.41 is the load balance IP (VIP).
> 
> I upgraded both source and target to 3.0.3 in case that would help (it
> hasn't).
> 
> SENDER (10.10.10.227)
> (syslog-ng.conf snippet)
> options {
>           time_reopen (2);
>           log_fifo_size (10000);
>           long_hostnames (off);
>           use_dns (no);
>           use_fqdn (no);
>           create_dirs (yes);
>           dir_perm (0755);
>           perm (0644);
>           chain_hostnames (no);
>           keep_hostname (yes);
>           stats_freq (3600);
>           log_msg_size (65535);
>           log_fifo_size (65536);
>         };
> 
> destination d_data { tcp("10.10.10.41" so_sndbuf(2094752)
> so_keepalive(yes)); };
> 
> (netstat)
> tcp        0      0 10.10.10.227:38370         10.10.10.41:514
> ESTABLISHED 2067/syslog-ng
> 
> 
> RECEIVER (10.10.10.31)
> (syslog-ng.conf snippet)
> source remote {
>         udp(ip(0.0.0.0) port(514) so_rcvbuf(1048576));
>         tcp(ip(0.0.0.0) port(514) max-connections(500)
> so_rcvbuf(1048576) so_keepalive(yes));
> };
> 
> (netstat)
> tcp        0      0 0.0.0.0:514                 0.0.0.0:*
> LISTEN      2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9501
> ESTABLISHED 2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9503
> ESTABLISHED 2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9499
> ESTABLISHED 2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9509
> ESTABLISHED 2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9511
> ESTABLISHED 2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9505
> ESTABLISHED 2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9507
> ESTABLISHED 2086/syslog-ng
> tcp        0      0 10.10.10.31:514            10.10.10.227:9513
> ESTABLISHED 2086/syslog-ng
> 

hmm.. if syslog-ng closes the connection immediately, the followings may
apply:

1) max-connections limit
2) tcp wrapper (e.g. /etc/hosts.allow and /etc/hosts.deny if enabled)
3) fd limit

you should try running strace on the running syslog-ng process and see
what it does when it rejects an incoming connection.

-- 
Bazsi