[syslog-ng]TCP reconnections randomly failing

[Xavier de Labouret]

Hello,

We are using syslog-ng with a central loghost. Our applications run on
remote servers, all their syslog logs being forwarded to the central
loghost with TCP forwarding.

However, we are facing a problem now with TCP reconnections. Sometimes,
we have to restart the syslog-ng daemon on our central loghost. And when
this happens, some clients fail to reconnect to the loghost.

The failing client syslog-ng complains about being unable to reconnect,
tries a variable number of reattempts, and then gives up. On the server
side, nothing is logged on these failed attempts.

We have no clue on why this fails on some servers and not on the others.
The maximum number of TCP connections on our loghost is set to 100, and
this number is never approached, by far. 

To have the TCP connection come back again with the remote server, we
must restart the client syslog-ng daemon.

Here are our logs on the client daemon:

-----------------8<---------------------

Mar 10 09:41:43 java-as02 syslog-ng[23311]: io.c: do_write: write()
failed (errno 32), Broken pipe
Mar 10 09:41:43 java-as02 syslog-ng[23311]: Connection broken, reopening
in 1 seconds
Mar 10 09:41:45 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds
Mar 10 09:41:46 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds
Mar 10 09:41:47 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds
Mar 10 09:41:49 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds
Mar 10 09:41:50 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds
Mar 10 09:41:51 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds
Mar 10 09:41:52 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds
Mar 10 09:41:53 java-as02 syslog-ng[23311]: Error connecting to remote
host (62.160.234.84), reattempting in 1 seconds

----------------->8---------------------

We can see 8 reconnection attempts, and then no more message.

Last precision: we are running syslog-ng 1.4.16 on Debian/Linux
platforms.

Here are my questions:

- could there be a weakness in the TCP reconnection procedure of
syslog-ng 1.4.16? 

- we don't think this comes from our network equipments. Maybe we could
get more traces of the syslog-ng behaviour in these cases. How could we
do it (options/tools)? 

- can we toggle some parameters now to make this less probably
happen (augmenting the reconnection delay...)?

- or maybe we should use syslog-ng 1.5.25?

Thank you in advance,

Xavier de Labouret