[syslog-ng] Connections not closing on syslog-ng 2.1.4 server

[Robert Nickel]

I have a busy syslog-ng server that is collecting a large volume of logs. This
problem is causing me grief and I am wondering if it is related to having
flow_control enabled on the host. Any help would be greatly appreciated.

The connection limit on the inbound tcp connection is slowly being exhausted
by multiple connections from each client slowly building until the server
stops working.

The source is configured as such:

  source s_remote { tcp(ip(0.0.0.0) port(5140) log_iw_size(204800) max-connections(2048)); };

The bulk of the logs are being sent to other syslog-ng servers running on the
same host. e.g.:

  destination child1 { tcp("127.0.0.1" port (5148)); };

All of the log lines have the same setup:

  log { 
    source (s_remote);
    filter(f_q1);
    destination(d_q1);
    flags(flow-control,final); 
  };

This setup is working very well with throughput in excess of 35000 messages
per second but the whole thing blows up every couple of days due to running
out of connections on the source.

Before it dies, for a couple of days, I see a number of these errors in the
logs:

  Number of allowed concurrent connections exceeded; num='2048', max='2048'

Usually, the actual number of connections is a couple of hundred above the
2048 number listed here. (netstat -an | fgrep -v ESTABLISHED | fgrep :5140).

The actual server count hovers around 1300 of which there are ~1000 actively
logging.

Options in use are:

  create_dirs(yes);
  dir_perm (0755);
  dns_cache_expire(28800);
  dns_cache (yes);
  flush_lines(10);
  flush_timeout(2048);
  frac_digits(3);
  keep_hostname(no);
  log_fetch_limit(100);
  log_fifo_size (2048);
  log_iw_size(100);
  long_hostnames(off);
  perm(0644);
  stats_freq(300);
  time_reopen(10);
  time_sleep(10);
  ts_format("iso");
  use_dns(yes);
  use_fqdn(no);
  use_time_recvd(no);

Thank you!
--Robert