[syslog-ng] syslog-ng 3.3.1 quits at reload

[Balazs Scheidler]

On Tue, 2011-11-08 at 16:08 -0500, Michael Hocke wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> 
> 
> Hi there,
> 
> I am having a problem with syslog-ng 3.3.1. Once in a while syslog-ng quits with a QUIT signal after it was asked to reload the configuration through a HUP signal. I have two instances of it running in separate Solaris 10 zones. Both are extremely busy and both crashed once within the last two weeks. By looking at the code in lib/mainloop.c I found the place where syslog-ng quits. It is within main_loop_reload_config_apply caused by a corrupt old configuration. Here is the backtrace with dbx:
> 
> - ------< SNIP >-------
> (dbx) where      
> current thread: t at 1
> =>[1] _kill(0x347f, 0x3, 0x0, 0xfef557d2, 0x80ef41c, 0xfed6217c), at 0xfeb7af85 
>   [2] main_loop_reload_config_apply(0x8047920, 0xfefad000, 0x8047908, 0xfef707f9, 0x80ef368, 0xfef92934, 0x0, 0x0, 0xfef706c9), at 0xfef70595 
>   [3] main_loop_io_worker_sync_call(0x0, 0x8047920, 0x0, 0x0, 0x0, 0x0), at 0xfef70742 
>   [4] iv_signal_event(0xfefad000, 0x8047960, 0x400, 0x796c746e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfef8d958 
>   [5] iv_event_raw_got_event(0xfefad018, 0x8047db0, 0xfe9, 0xfefaa468, 0x2, 0x1), at 0xfef8d54a 
>   [6] iv_main(0xfefad180, 0xfef9291e, 0x804c2e8, 0x0, 0x804c7b0, 0x8047e1c), at 0xfef8ac82 
>   [7] main_loop_run(0x8049cd7, 0x8047e44, 0x8047e48, 0x8047e2c, 0x8047e30, 0x8049390), at 0xfef71259 
>   [8] main(0x1, 0x8047e60, 0x8047e70), at 0x80490fa 
> - ------< SNIP >-------

Hmm... syslog-ng only sends itself a SIGQUIT if the reload goes really
bad.

At reload time, syslog-ng tries to:
  * deinit the old configuration (close stuff, etc)
  * init the new configuration (open stuff, etc)

If the init of the 2nd configuration fails, it tries to revert back to
the old by initializing that again. And this fails. In this case
syslog-ng has no configuration to operate on (e.g. has no clue where tto
listen for messages and where to write them) and kills itself.

The issue is worse as it has no place to write this information as a
troubleshooting aid if running in the background, as it has no console
at this point. Perhaps you could run syslog-ng in the foreground,
redirect all internal logs to stderr using the -e switch and see what
causes trouble?

I don't see why the config can't be initialized, it seems to be pretty
basic. Perhaps one of the device files you operate on (/dev/sysmsg)
can't be opened in some cases. Maybe a bug causes syslog-ng to keep that
file opened and trying it open it again? (I've checked this case, and
syslog-ng seems to be closing the file descriptor just fine).

Maybe it's related to /dev/log, which again may not be opened multiple
times.

-- 
Bazsi