[syslog-ng] syslog-ng hanging?

[Chris Whipple]

I'm seeing the same problem. When the connection to a remote server that we
are sending logs to via TCP is broken, presumably from a temporary DNS or
network error, the server starts stacking /dev/log connections till there
are hundreds of them and any program that attempts to write log entries
hangs, including logins.

I can recreate the problem at will by sending logs from a test sender to a
test collector then blocking traffic with IPTables to simulate a network
issue. The sending server never recovers the connections even after turning
the IPTables back off. Syslog-ng has to be restarted on the sender to clear
up the issue. If not restarted, it will eventually lockup the server. I've
had to write a "baby-sitter" process to watch for logging to stop then
automatically restart NG when necessary on critical servers.

I've already tried several variations of the keep-alive, tcp-keep-alive,
log_fifo_size, etc to no avail.

We were running 1.6.0rc3 but upgraded to 1.6.10 on a few servers in hopes
of correcting it, but it hasn't.

Here are the simplified configs I'm using on the test servers:

-----------------------------------------------------------------------------------------------------
#Syslog-NG Test Sending Server
options {
       use_dns(no);
       use_fqdn(yes);
       sync(0);
       stats(3600);
       time_reopen(10);
       log_fifo_size(4096);
       log_msg_size(8192);
};
source s_local  { internal();
                unix-stream("/dev/log"
                        keep-alive(yes)
                        max-connections(100));
                file("/proc/kmsg"); };
destination d_collector  { tcp("testcollector.hertz.com" port(514)
                         tcp-keep-alive(yes)); };
filter f_loc2   { facility(local2); };
log { source(s_local); filter(f_loc2); destination(d_collector); };
-----------------------------------------------------------------------------------------------------
#Syslog-NG Test Collector Server
options {
        use_dns(no);
       sync(0);
       stats(3600);
       time_reopen(10);
       log_fifo_size(4096);
       log_msg_size(8192);
};
source s_local  { internal();
                 unix-stream("/dev/log"
                         keep-alive(yes)
                         max-connections(100));
                 file("/proc/kmsg"); };
source s_tcp    { tcp(port(514) keep-alive(yes)
                        tcp-keep-alive(yes));
                        max-connections(1000)); };
filter f_loc2           { facility(local2); };
destination d_loc2      { file("/tmp/test-loc2.log"); };
log {source(s_tcp); filter(f_loc2); destination(d_loc2); };
-----------------------------------------------------------------------------------------------------


Thank you,

Chris Whipple
Sr. Security Analyst
Unix Security Group
The Hertz Corporation
5601 NW Expressway
Oklahoma City, OK 73132, USA
cwhipple at hertz.com



---------------
This message (including attachments) may contain information that is 
privileged, confidential or protected from disclosure.  If you are not the 
intended recipient, you are hereby notified that dissemination, 
disclosure, copying, distribution or use of this message or any 
information contained in it is strictly prohibited.  If you have received 
this message in error, please immediately notify the sender by reply 
e-mail and delete this message from your computer.  Although we have taken 
steps to ensure that this e-mail and attachments are free from any virus, 
we advise that in keeping with good computing practice the recipient 
should ensure they are actually virus free.
---------------