increasing number of TCP connections from same number of remote hosts
Hi, It's syslog-ng v2.0.7 on RHEL4, compiled from source. The system is a log server, it receives logs via TCP from various clients. After restarting syslog-ng, netstat -t shows that each remote host open only one TCP connection to the server, which is normal. The problem is, that the number of established TCP connections is increasing constantly, but the number of clients is the same. For example netstat -t --numeric-ports | grep fmx23 now shows: tcp 0 0 barapp1:514 fmx23.freemail.privat:52391 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50852 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50172 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:59367 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50979 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:55828 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:53013 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50038 ESTABLISHED Why is that? AFAIK there should be only one established connection per client. Is it a server or client problem? Thanks, Daniel
On Thu, 2008-01-24 at 08:39 +0100, Nagy Daniel wrote:
Hi,
It's syslog-ng v2.0.7 on RHEL4, compiled from source. The system is a log server, it receives logs via TCP from various clients.
After restarting syslog-ng, netstat -t shows that each remote host open only one TCP connection to the server, which is normal. The problem is, that the number of established TCP connections is increasing constantly, but the number of clients is the same.
For example netstat -t --numeric-ports | grep fmx23 now shows:
tcp 0 0 barapp1:514 fmx23.freemail.privat:52391 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50852 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50172 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:59367 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50979 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:55828 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:53013 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50038 ESTABLISHED
Why is that? AFAIK there should be only one established connection per client. Is it a server or client problem?
Hmm.. strange, it should not do that. It is probably a client problem, can you check if it is indeed the syslog-ng process that opens these connections? Can you see messages like this on the client: msg_error("Connection broken", evt_tag_int("time_reopen", self->time_reopen), NULL); If you enable verbose logging, somewhat more information should be displayed about the reasons why syslog-ng reconnects. -- Bazsi
Hello, On the client fmx23 there is only one TCP connection shown by netstat, however there are several connection broken messages: Jan 23 17:16:19 fmx23 syslog-ng[836]: EOF occurred while idle; fd='12' Jan 23 17:16:19 fmx23 syslog-ng[836]: Connection broken; time_reopen='60' The clients have syslog-ng 2.0.3. Daniel Balazs Scheidler wrote:
On Thu, 2008-01-24 at 08:39 +0100, Nagy Daniel wrote:
Hi,
It's syslog-ng v2.0.7 on RHEL4, compiled from source. The system is a log server, it receives logs via TCP from various clients.
After restarting syslog-ng, netstat -t shows that each remote host open only one TCP connection to the server, which is normal. The problem is, that the number of established TCP connections is increasing constantly, but the number of clients is the same.
For example netstat -t --numeric-ports | grep fmx23 now shows:
tcp 0 0 barapp1:514 fmx23.freemail.privat:52391 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50852 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50172 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:59367 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50979 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:55828 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:53013 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50038 ESTABLISHED
Why is that? AFAIK there should be only one established connection per client. Is it a server or client problem?
Hmm.. strange, it should not do that. It is probably a client problem, can you check if it is indeed the syslog-ng process that opens these connections?
Can you see messages like this on the client:
msg_error("Connection broken", evt_tag_int("time_reopen", self->time_reopen), NULL);
If you enable verbose logging, somewhat more information should be displayed about the reasons why syslog-ng reconnects.
Hello, It was a firewall problem... There is a firewall between the server and the clients. The firewall breaks TCP connections after a specified idle time. That's why the clients re-establish the connections. Would it possible to include TCP keepalive feature in syslog-ng? This way the server could check the established connections regurarly and tear down the broken ones. Daniel Nagy Daniel wrote:
Hello,
On the client fmx23 there is only one TCP connection shown by netstat, however there are several connection broken messages:
Jan 23 17:16:19 fmx23 syslog-ng[836]: EOF occurred while idle; fd='12' Jan 23 17:16:19 fmx23 syslog-ng[836]: Connection broken; time_reopen='60'
The clients have syslog-ng 2.0.3.
Daniel
Balazs Scheidler wrote:
On Thu, 2008-01-24 at 08:39 +0100, Nagy Daniel wrote:
Hi,
It's syslog-ng v2.0.7 on RHEL4, compiled from source. The system is a log server, it receives logs via TCP from various clients.
After restarting syslog-ng, netstat -t shows that each remote host open only one TCP connection to the server, which is normal. The problem is, that the number of established TCP connections is increasing constantly, but the number of clients is the same.
For example netstat -t --numeric-ports | grep fmx23 now shows:
tcp 0 0 barapp1:514 fmx23.freemail.privat:52391 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50852 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50172 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:59367 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50979 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:55828 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:53013 ESTABLISHED tcp 0 0 barapp1:514 fmx23.freemail.privat:50038 ESTABLISHED
Why is that? AFAIK there should be only one established connection per client. Is it a server or client problem? Hmm.. strange, it should not do that. It is probably a client problem, can you check if it is indeed the syslog-ng process that opens these connections?
Can you see messages like this on the client:
msg_error("Connection broken", evt_tag_int("time_reopen", self->time_reopen), NULL);
If you enable verbose logging, somewhat more information should be displayed about the reasons why syslog-ng reconnects.
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Thu, 2008-01-24 at 14:34 +0100, Nagy Daniel wrote:
Hello,
It was a firewall problem... There is a firewall between the server and the clients. The firewall breaks TCP connections after a specified idle time. That's why the clients re-establish the connections.
Would it possible to include TCP keepalive feature in syslog-ng? This way the server could check the established connections regurarly and tear down the broken ones.
It is included, but you need to explicitly enable it using the so_keepalive(yes) option. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Nagy Daniel