[syslog-ng] Listen backlog issues?

[Scheidler, Balázs]

Hi,

I've just submitted this pull request:
https://github.com/balabit/syslog-ng/pull/1211

Hope this resolves your problem.

Cheers,
Bazsi

-- 
Bazsi

On Mon, Sep 19, 2016 at 10:51 AM, Claus Albøge <ca at tdchosting.dk> wrote:

> Hi,
>
> I have ~6000 hosts sending syslog-TLS to a syslog-ng server. I’m seeing a
> lot of reconnects from the clients, and on the syslog-ng servers I get the
> following messages:
>
>   TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending
> cookies.  Check SNMP counters.
>
> netstat -s show the following:
>
> $ netstat -s | grep -i listen
>     10480620 times the listen queue of a socket overflowed
>     11144791 SYNs to LISTEN sockets dropped
>
> A few resources suggest this could be due to the applications socket
> listen backlog.
>
> Starting syslog-ng with strace reveals the following:
>
>   30067 listen(5, 255)                    = 0
>   30067 listen(11, 255)                   = 0
>   30067 listen(12, 255)                   = 0
>   30067 listen(17, 255)                   = 0
>
> It seems like the backlog is hardcoded to 255 in the source.
>
> A bit more info:
>
> Saving the output from: “netstat -n | grep -c SYN_RECV” to a file a few
> times per second, shows the following:
>
> 2016-09-19 09:54:14.767469015: 0
> 2016-09-19 09:54:14.906638364: 0
> 2016-09-19 09:54:15.048591252: 0
> 2016-09-19 09:54:15.184325070: 0
> 2016-09-19 09:54:15.324150368: 0
> 2016-09-19 09:54:15.459749187: 0
> 2016-09-19 09:54:15.587983284: 0
> 2016-09-19 09:54:15.722558975: 42
> 2016-09-19 09:54:15.873601766: 256
> 2016-09-19 09:54:16.020145083: 247
> 2016-09-19 09:54:16.177231109: 178
> 2016-09-19 09:54:16.340875439: 178
> 2016-09-19 09:54:16.488506916: 178
> 2016-09-19 09:54:16.637815500: 176
> 2016-09-19 09:54:16.790781389: 149
> 2016-09-19 09:54:16.950254517: 141
> 2016-09-19 09:54:17.125316109: 131
> 2016-09-19 09:54:17.286839687: 131
> 2016-09-19 09:54:17.445583267: 127
> 2016-09-19 09:54:17.598144758: 127
> 2016-09-19 09:54:17.747967473: 124
> 2016-09-19 09:54:17.905359412: 2
> 2016-09-19 09:54:18.062665358: 2
> 2016-09-19 09:54:18.239973822: 0
> 2016-09-19 09:54:18.398021514: 0
> 2016-09-19 09:54:18.548074304: 0
>
> This matches the entry in /var/log/messages
>
> Sep 19 09:54:15 log03 kernel: [1116811.081666] TCP: request_sock_TCP:
> Possible SYN flooding on port 6514. Sending cookies.  Check SNMP counters.
>
> Are there any tunings in the syslog-ng configuration I need to implement,
> or do I have to compile syslog-ng myself with a larger listen backlog?
> Perhaps this should be configurable?
>
> My source definition for syslog-TLS is like this:
>
> source s_syslog_tls {
>   syslog(
>     ip(0.0.0.0)
>     port(6514)
>     log_fetch_limit(100)
>     log_iw_size(11000000)
>     max-connections(10000)
>     transport("tls")
>     tls(
>       key-file("/etc/pki/tls/certs/logserver.key")
>       cert-file("/etc/pki/tls/certs/logserver.crt")
>       peer-verify(optional-untrusted)
>     )
>   );
> };
>
> $ syslog-ng -V
> syslog-ng 3.8.1
> Installer-Version: 3.8.1
> Revision:
> Module-Directory: //usr/lib64/syslog-ng
> Module-Path: //usr/lib64/syslog-ng
> Available-Modules: disk-buffer,sdjournal,afstomp,json-plugin,cryptofuncs,
> graphite,cef,kvformat,add-contextual-data,dbparser,
> pseudofile,curl,csvparser,syslogformat,confgen,afsocket,
> afuser,date,linux-kmsg-format,system-source,basicfuncs,
> afamqp,affile,afprog
> Enable-Debug: off
> Enable-GProf: off
> Enable-Memtrace: off
> Enable-IPv6: on
> Enable-Spoof-Source: on
> Enable-TCP-Wrapper: on
> Enable-Linux-Caps: off
>
> syslog-ng-3.8.1-1.el7.centos.x86_64 from https://copr-be.cloud.
> fedoraproject.org/results/czanik/syslog-ng38/epel-7-x86_64/
>
> Please let me know if more info is needed.
>
>
> /Claus A
>
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160919/54e55d61/attachment.htm