[syslog-ng] syslog hangup

[Balazs Scheidler]

On Fri, 2006-07-28 at 12:38 +0200, Vincent Régnard wrote:
> Hi,
> 
> Since at least a couple of month now, we've been experiencing some 
> strange troubles on some of our routers which we now associate to a 
> syslog-ng (or a syslog-ng internal facility) hangup. The version of 
> syslog ng we use is 1.6.8 on different linux kernel 2.4.31 and 2.6.8.
> 
> I read some posts on this list 
> (https://lists.balabit.hu/pipermail/syslog-ng/2006-May/008784.html) 
> establishing some similar problems.
> 
> The conditions in which the problem occur are not clear to us. There are 
>   obviously different sircumstances that lead syslog-ng to hangup.
> 
> We first noticed the problem on a server where some users where not able 
> to login, certainly because of the impossibility to write to the 
> /dev/log socket. Reloading syslog-ng when we had an active shell on the 
> server corrected the problem. For this server we "solved" the problem by 
> adding a cron to check the syslog activity and reload syslog if needed. 
> This is not a nice soltution, but it avoids hard reboot of the server.
> 
> More recently we realized that some routers where not logging some 
> (iptables firewall) events sent to syslog, when in the same time, log 
> from other daemons where treated correctly, again reloading syslog-ng 
> fixes the problem untill the problem randmly accurs again. I am 
> presently studdying the way these log messages are sent to syslog to 
> understand this trouble better.
> 
> We are tracking the causes of such an annoying behaviour without succes 
> untill now. First of all we would like to understand what is happening 
> in syslog-ng itselfs, at what level is this hangup ? kernel ? syslog ? 
> is it related to /dev/log socket ? Maybe some experts or syslog 
> devloppers can send us some hints ? Is it related to the kernel 
> environement ? /proc ? udev ? Or is it possible that another daemon is 
> responsible for this syslog hangup.
> 
> Apparently the problem is also present in newer releases in the 1.6.X 
> branch according to the posts on the list, I checked the branch 
> changelogs without seeing anything on that. Has some work been devoted 
> to fix this kind of trouble in more recent branches (1.9 and 2.0) ?
> 
> We are planning to develop a daemon to monitor syslog-ng and reload the 
> service in case of hangup. If some of you already performed some work in 
> that direction, we would be glad to share the effort or learn the best 
> and more efficient way proceed.

The only hang cause I know about is not really a syslog-ng issue, at
least not fixable in syslog-ng alone (although I've already tried to
work it around).

This problem is related to reading the /proc/kmsg special file, as if
multiple processes poll /proc/kmsg, one of them might block as the
kernel does not support non-blocking I/O on /proc/kmsg.

This is usually caused by:

1) klogd and syslog-ng running on the same host, syslog-ng
referencing /proc/kmsg

2) two syslog-ng instances running for some reason (started two times
because of lost pidfiles)

3) one syslog-ng having more than a single /proc/kmsg source


-- 
Bazsi