[syslog-ng] Listen backlog issues?

[Claus Albøge]

Hi,

I have ~6000 hosts sending syslog-TLS to a syslog-ng server. I’m seeing a lot of reconnects from the clients, and on the syslog-ng servers I get the following messages: 

  TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies.  Check SNMP counters.

netstat -s show the following:

$ netstat -s | grep -i listen
    10480620 times the listen queue of a socket overflowed
    11144791 SYNs to LISTEN sockets dropped

A few resources suggest this could be due to the applications socket listen backlog.

Starting syslog-ng with strace reveals the following: 

  30067 listen(5, 255)                    = 0
  30067 listen(11, 255)                   = 0
  30067 listen(12, 255)                   = 0
  30067 listen(17, 255)                   = 0

It seems like the backlog is hardcoded to 255 in the source.

A bit more info:

Saving the output from: “netstat -n | grep -c SYN_RECV” to a file a few times per second, shows the following:

2016-09-19 09:54:14.767469015: 0
2016-09-19 09:54:14.906638364: 0
2016-09-19 09:54:15.048591252: 0
2016-09-19 09:54:15.184325070: 0
2016-09-19 09:54:15.324150368: 0
2016-09-19 09:54:15.459749187: 0
2016-09-19 09:54:15.587983284: 0
2016-09-19 09:54:15.722558975: 42
2016-09-19 09:54:15.873601766: 256
2016-09-19 09:54:16.020145083: 247
2016-09-19 09:54:16.177231109: 178
2016-09-19 09:54:16.340875439: 178
2016-09-19 09:54:16.488506916: 178
2016-09-19 09:54:16.637815500: 176
2016-09-19 09:54:16.790781389: 149
2016-09-19 09:54:16.950254517: 141
2016-09-19 09:54:17.125316109: 131
2016-09-19 09:54:17.286839687: 131
2016-09-19 09:54:17.445583267: 127
2016-09-19 09:54:17.598144758: 127
2016-09-19 09:54:17.747967473: 124
2016-09-19 09:54:17.905359412: 2
2016-09-19 09:54:18.062665358: 2
2016-09-19 09:54:18.239973822: 0
2016-09-19 09:54:18.398021514: 0
2016-09-19 09:54:18.548074304: 0

This matches the entry in /var/log/messages

Sep 19 09:54:15 log03 kernel: [1116811.081666] TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies.  Check SNMP counters.

Are there any tunings in the syslog-ng configuration I need to implement, or do I have to compile syslog-ng myself with a larger listen backlog? Perhaps this should be configurable? 

My source definition for syslog-TLS is like this:

source s_syslog_tls {
  syslog(
    ip(0.0.0.0)
    port(6514)
    log_fetch_limit(100)
    log_iw_size(11000000)
    max-connections(10000)
    transport("tls")
    tls(
      key-file("/etc/pki/tls/certs/logserver.key")
      cert-file("/etc/pki/tls/certs/logserver.crt")
      peer-verify(optional-untrusted)
    )
  );
};

$ syslog-ng -V
syslog-ng 3.8.1
Installer-Version: 3.8.1
Revision:
Module-Directory: //usr/lib64/syslog-ng
Module-Path: //usr/lib64/syslog-ng
Available-Modules: disk-buffer,sdjournal,afstomp,json-plugin,cryptofuncs,graphite,cef,kvformat,add-contextual-data,dbparser,pseudofile,curl,csvparser,syslogformat,confgen,afsocket,afuser,date,linux-kmsg-format,system-source,basicfuncs,afamqp,affile,afprog
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: off

syslog-ng-3.8.1-1.el7.centos.x86_64 from https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-7-x86_64/

Please let me know if more info is needed.


/Claus A