[syslog-ng] syslog-ng crashed

[Balazs Scheidler]

On Mon, 2011-01-24 at 22:12 -0800, Subrahmanyam Manuguri wrote:
> Hi,
> 
> I am using the following version of syslog-ng server on a PowerPC
> based system running Montavista Linux.
> 
> 
> syslog-ng --version
> syslog-ng 3.0.3
> Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#08c9bf9d98e4e021756adc12c42605840140ba8b
> Compile-Date: Jul 23 2009 14:23:17
> Enable-Threads: off
> Enable-Debug: off
> Enable-GProf: off
> Enable-Memtrace: off
> Enable-Sun-STREAMS: off
> Enable-Sun-Door: off
> Enable-IPv6: on
> Enable-Spoof-Source: off
> Enable-TCP-Wrapper: on
> Enable-SSL: on
> Enable-SQL: off
> Enable-Linux-Caps: on
> Enable-Pcre: on
> 
> Of late, I am running into the syslog-ng crashes during the boot up of
> the system. At the boot up time the number of messages are
> significantly more. I was able to collect the stack trace.
> 
> (gdb) bt
> #0  0x00000000 in ?? ()
> #1  0x10024d0c in log_pipe_queue ()
> #2  0x10024c1c in log_multiplexer_queue ()
> #3  0x100304a8 in log_pipe_queue ()
> #4  0x10030440 in log_source_group_queue ()
> #5  0x1000f218 in log_pipe_queue ()
> #6  0x1000f1b4 in log_pipe_forward_msg ()
> #7  0x1000f218 in log_pipe_queue ()
> #8  0x1000f1b4 in log_pipe_forward_msg ()
> #9  0x10045030 in log_pipe_queue ()
> #10 0x10044f08 in log_source_queue ()
> #11 0x100318d0 in log_pipe_queue ()
> #12 0x1003185c in log_reader_handle_line ()
> #13 0x10031bf4 in log_reader_fetch_log ()
> #14 0x100315b8 in log_reader_fd_dispatch ()
> #15 0x0ff132e4 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
> #16 0x0ff16c9c in ?? () from /usr/lib/libglib-2.0.so.0
> #17 0x0ff17248 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
> #18 0x10006200 in main_loop_run ()
> #19 0x100069c4 in main ()
> (gdb)
> 
> The configuration i am running with is
> 
> 
> options {
>        time_sleep(10);
>        log_fetch_limit(250);
>        log_fifo_size(2000);
>        keep_timestamp(no);
>        dns_cache(no);
>        dns_cache_hosts(/etc/hosts);
>        use_dns(persist_only);
>        flush_timeout(200);
>        perm(0644);
>        stats_freq(3600);
>        stats_level(0);
>        ts_format(iso);
>        time_reopen(10);
>        create_dirs(yes);
>        dir_perm(755);
> };
> 
> Can somebody please help me in tweaking the right parameters to avoid
> the problem (if that solves the problem)?
> 
> If this is a known problem, is there a patch available?
> 
> Appreciate your help very much.

Hmm.. your version of syslog-ng is quite outdated. I have some vague
remembering that there was a similar bug fixed, but the only clue I've
found was before 3.0.3 got released:

    [logprocess] fixed log_process_queue to handle NULL "next" pointers (fixes: pub#49)
    
    With a configuration which had a filter() but no destination, it can happen
    that the next pointer of a filter() node points to nowhere.  This patch
    handles that case without causing a crash.
    
    Reported-By: Jan Schaumann

The bug seems to be related to your configuration file though, since it seems 
to be dereferencing a NULL function pointer while going "towards" the destination 
side of your configuration. It is probably triggered by a single message, which 
occurs during bootup, and doesn't occur later.

Can you try 3.0.10 (the latest in 3.0.x, or perhaps an even newer version) in 
the same environment? If it crashes, then debug symbols would help even more.

Thanks for the report.

-- 
Bazsi