[syslog-ng] syslog-ng suddenly stops logging

[Richard Vigeant]

On 24-Jun-08, at 3:37 AM, Balazs Scheidler wrote:

> On Thu, 2008-06-19 at 15:54 -0400, Richard Vigeant wrote:
>> Hi,
>>
>>
>> I have a configuration where several nodes send all log messages to a
>> central server. The
>> applications on remote nodes send their logs locally either via UDP  
>> or
>> a unix socket. The
>> syslog-ng running on remote nodes simply pick up all log messages  
>> from
>> all sources, i.e. TCP, UDP,
>> /proc/kmsg, /dev/log and internal, and transmit all messages to the
>> central server uisng TCP. The
>> remote node's config file follows.
>>
>>
>> We've been having intermittent problems where the central server  
>> would
>> suddenly stop logging messages
>> from certain nodes. We noticed that very often restarting syslog-ng  
>> on
>> the central server would fix
>> the condition and logging would carry on.
>>
>>
>> Howver I discovered a new rare case where restarting the central
>> syslog-ng didn't work. I found out
>> by doing a tcpdump that the remote syslog-ng was not sending the log
>> messages. I have done an strace
>> on the remote syslog-ng and it shows that nothing happens after a
>> message has been "recvfrom()" or
>> "read()". Then I have restarted syslog-ng and things went back to
>> normal. In the 2nd strace we can see
>> that there is a "write()" after the "read()".
>>
>
> I might be guessing here as I don't really know which fd is which,  
> but I
> think you've ran into an issue that some others have experienced
> previously.
>
> In the case when the traffic does not work, syslog-ng is correctly
> polling fd 8 for output, I assumed that fd 8 is the fd of the  
> connection
> to the server. (it is in the 2nd strace dump).
>
> So syslog-ng is polling for writing out on fd 8, but the poll system
> call does not indicate writability. This usually means that the tcp()
> window is full, the server does not accept new data.
>
> State based firewalls often drop inactive connections after a period  
> of
> time and in case packets arrive for a connection for which no state
> exists, packets are dropped.
>
> Do you have a firewall between the client and the server?

No firewall. Clients and server are all on the same LAN. This is one  
of our local QA environment.

Note that I have seen similar cases where the problem occurred on the  
server and the output is a file. However I can't currently reproduce it.
>
>
> -- 
> Bazsi
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>

Richard Vigeant
Senior Software Developper
Vantrix

T +1 514-866-1717
M +1 438-275-6796
F +1 514-866-6868

richard.vigeant at vantrix.com
www.vantrix.com