[syslog-ng] syslog-ng suddenly stops logging

[Balazs Scheidler]

On Thu, 2008-06-19 at 15:54 -0400, Richard Vigeant wrote:
> Hi,
> 
> 
> I have a configuration where several nodes send all log messages to a
> central server. The 
> applications on remote nodes send their logs locally either via UDP or
> a unix socket. The
> syslog-ng running on remote nodes simply pick up all log messages from
> all sources, i.e. TCP, UDP, 
> /proc/kmsg, /dev/log and internal, and transmit all messages to the
> central server uisng TCP. The 
> remote node's config file follows.
> 
> 
> We've been having intermittent problems where the central server would
> suddenly stop logging messages 
> from certain nodes. We noticed that very often restarting syslog-ng on
> the central server would fix
> the condition and logging would carry on.
> 
> 
> Howver I discovered a new rare case where restarting the central
> syslog-ng didn't work. I found out 
> by doing a tcpdump that the remote syslog-ng was not sending the log
> messages. I have done an strace 
> on the remote syslog-ng and it shows that nothing happens after a
> message has been "recvfrom()" or 
> "read()". Then I have restarted syslog-ng and things went back to
> normal. In the 2nd strace we can see 
> that there is a "write()" after the "read()".
> 

I might be guessing here as I don't really know which fd is which, but I
think you've ran into an issue that some others have experienced
previously.

In the case when the traffic does not work, syslog-ng is correctly
polling fd 8 for output, I assumed that fd 8 is the fd of the connection
to the server. (it is in the 2nd strace dump). 

So syslog-ng is polling for writing out on fd 8, but the poll system
call does not indicate writability. This usually means that the tcp()
window is full, the server does not accept new data. 

State based firewalls often drop inactive connections after a period of
time and in case packets arrive for a connection for which no state
exists, packets are dropped.

Do you have a firewall between the client and the server?

-- 
Bazsi