[Bug 97] New: syslog-ng crashes after SIGHUP
https://bugzilla.balabit.com/show_bug.cgi?id=97 Summary: syslog-ng crashes after SIGHUP Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: zhengxiang.pan@alcatel-lucent.com Type of the Report: bug Estimated Hours: 0.0 Syslog-ng 3.0.6 crashes with following trace: Program terminated with signal 11, Segmentation fault. [New process 6311] #0 0x08063558 in log_source_group_queue (s=0x8ca6e00, msg=0x8caecc0, path_options=0xbfffaf18) at sgroup.c:98 98 (*self->processed_messages)++; (gdb) bt #0 0x08063558 in log_source_group_queue (s=0x8ca6e00, msg=0x8caecc0, path_options=0xbfffaf18) at sgroup.c:98 #1 0x08070580 in log_source_queue (s=0x8cadc78, msg=0x8caecc0, path_options=0xbfffaf74) at logpipe.h:121 #2 0x08063e80 in log_reader_fd_dispatch (source=0x8cadd10, callback=0, user_data=0x0) at logpipe.h:121 #3 0x08080081 in g_main_context_dispatch () #4 0x08083103 in g_main_context_iterate () #5 0x0808327f in g_main_context_iteration () #6 0x0804b52f in main_loop_run (cfg=0xbfffb0e8) at main.c:163 #7 0x0804bbcd in main (argc=147613904, argv=0xbfffb110) at main.c:443 The problem happens from version 3.0.6 ( I did not test other version lower than that) until the lastest 3.1.3. I guess what happens is that when SIGHUP is recevied, some connections are persisted, while the persistent connections points to the already-freed pipe (via pipe_next), if there are messages coming before the freed-pipes are replaced, the messages are forwarded to the freed-pipe handler. For version 3.1.3, in logpipe.c, log_pipe_forward_msg() may add check the pipe_next flags, however this may be not enough. log_pipe_forward_msg(LogPipe *self, LogMessage *msg, const LogPathOptions *path_options) { if (self->pipe_next && self->flags&PIF_INITIALIZED) { log_pipe_queue(self->pipe_next, msg, path_options); -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=97 --- Comment #1 from Balazs Scheidler <bazsi@balabit.hu> 2010-12-02 08:58:43 --- This is odd. There's code to set the owner & pipe_next pointers in persistent connections, see the function afsocket_sc_set_owner() which is called from afsocket_sd_init() for all recovered connections. Is there a description how this crash happens? Reproducing it here would help a lot to diagnose the problem. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=97 --- Comment #2 from zhengxiang pan <zhengxiang.pan@alcatel-lucent.com> 2010-12-02 15:54:45 --- Created an attachment (id=25) --> (https://bugzilla.balabit.com/attachment.cgi?id=25) core file -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=97 --- Comment #3 from zhengxiang pan <zhengxiang.pan@alcatel-lucent.com> 2010-12-02 16:08:05 --- What I did is multiple times of syslog-ng.conf modification following "killall -SIGHUP syslog-ng". I could easily reproduce it with my script, but which requires many special setup. I realized that the core trace I posted originally may not correct as the core file and binary may not match. I attached core file of syslog-ng 3.1.3 for your analysis. The binary of syslog-ng 3.1.3 is too big to attach, but I use the following to build on Ubuntu 9.0.4: ./configure --prefix=<dir> --enable-ipv6 --enable-mixed-linking --enable-debug The core trace below: Program terminated with signal 11, Segmentation fault. [New process 28303] #0 0x0806b4fe in log_source_group_queue (s=0x8afec98, msg=0x8afb068, path_options=0xbfa4f0f8) at sgroup.c:98 98 (*self->processed_messages)++; (gdb) bt #0 0x0806b4fe in log_source_group_queue (s=0x8afec98, msg=0x8afb068, path_options=0xbfa4f0f8) at sgroup.c:98 #1 0x08055070 in log_pipe_queue (s=0x8afec98, msg=0x8afb068, path_options=0xbfa4f0f8) at logpipe.h:121 #2 0x08055038 in log_pipe_forward_msg (self=0x8afeb70, msg=0x8afb068, path_options=0xbfa4f0f8) at logpipe.c:72 #3 0x08055070 in log_pipe_queue (s=0x8afeb70, msg=0x8afb068, path_options=0xbfa4f0f8) at logpipe.h:121 #4 0x08055038 in log_pipe_forward_msg (self=0x8ae57a8, msg=0x8afb068, path_options=0xbfa4f0f8) at logpipe.c:72 #5 0x0807b9f7 in log_pipe_queue (s=0x8ae57a8, msg=0x8afb068, path_options=0xbfa4f0f8) at logpipe.h:121 #6 0x0807b96f in log_source_queue (s=0x8ae7898, msg=0x8afb068, path_options=0xbfa4f164) at logsource.c:186 #7 0x0806c438 in log_pipe_queue (s=0x8ae7898, msg=0x8afb068, path_options=0xbfa4f164) at logpipe.h:121 #8 0x0806c405 in log_reader_handle_line (self=0x8ae7898, line=0x8ae5890 "<46>syslog-ng[28303]: Closing log transport fd; fd='16'\n", length=57, saddr=0x8b00730, parse_flags=0) at logreader.c:295 #9 0x0806c65e in log_reader_fetch_log (self=0x8ae7898, proto=0x8ae5830) at logreader.c:374 #10 0x0806c1d8 in log_reader_fd_dispatch (source=0x8ae7910, callback=0, user_data=0x0) at logreader.c:228 #11 0x0808c6a1 in g_main_context_dispatch () #12 0x0808f723 in g_main_context_iterate () #13 0x0808f89f in g_main_context_iteration () #14 0x0804b7ba in main_loop_run (cfg=0xbfa4f400) at main.c:166 #15 0x0804bfb2 in main (argc=1, argv=0xbfa4f4a4) at main.c:480 (gdb) p s (LogPipe *) 0x8afec98 (gdb) p *s $1 = {ref_cnt = 145684656, flags = 0, cfg = 0x0, pipe_next = 0x8afaae8, queue = 0x806b480 <log_source_group_queue>, init = 0x806b128 <log_source_group_init>, deinit = 0x806b399 <log_source_group_deinit>, free_fn = 0x806b52b <log_source_group_free>, notify = 0} (gdb) up #2 0x08055038 in log_pipe_forward_msg (self=0x8afeb70, msg=0x8afb068, path_options=0xbfa4f0f8) at logpipe.c:72 72 log_pipe_queue(self->pipe_next, msg, path_options); (gdb) p *self <------------------------------------------------------ self=0x8afeb70 $2 = {ref_cnt = 1, flags = 0, cfg = 0x0, pipe_next = 0x8afec98, queue = 0x805500c <log_pipe_forward_msg>, init = 0x8074784 <afsocket_sd_init>, deinit = 0x8074c02 <afsocket_sd_deinit>, free_fn = 0x8074e15 <afsocket_sd_free>, notify = 0x8074dae <afsocket_sd_notify>} (gdb) up #4 0x08055038 in log_pipe_forward_msg (self=0x8ae57a8, msg=0x8afb068, path_options=0xbfa4f0f8) at logpipe.c:72 72 log_pipe_queue(self->pipe_next, msg, path_options); (gdb) p *self $3 = {ref_cnt = 2, flags = 1, cfg = 0x0, pipe_next = 0x8afeb70, queue = 0x805500c <log_pipe_forward_msg>, init = 0x8073943 <afsocket_sc_init>, deinit = 0x8073c38 <afsocket_sc_deinit>, free_fn = 0x8073e23 <afsocket_sc_free>, gdb) up #6 0x0807b96f in log_source_queue (s=0x8ae7898, msg=0x8afb068, path_options=0xbfa4f164) at logsource.c:186 186 log_pipe_queue(s->pipe_next, msg, &local_options); (gdb) p *s $4 = {ref_cnt = 2, flags = 1, cfg = 0x0, pipe_next = 0x8ae57a8, queue = 0x807b670 <log_source_queue>, init = 0x806c72f <log_reader_init>, deinit = 0x806c8b5 <log_reader_deinit>, free_fn = 0x806c915 <log_reader_free>, notify = 0}
From the trace, you can see that both pipe 0x8afec98 and 0x8afeb70 has been freed, pipe 0x8ae57a8 and 0x8ae7898 are persistent socket and reader pipes, which still refer to the freed pipes.
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=97 --- Comment #4 from Balazs Scheidler <bazsi@balabit.hu> 2010-12-03 22:58:56 --- Hmm. it is possible that the lingering connection belongs to a source that is not present in the new configuration? e.g. steps to reproduce: 1) open a connection to syslog-ng 2) remove that source 3) SIGHUP 4) generate messages on the still open connection. With this I seem to have produced a crash with a similar backtrace. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=97 Balazs Scheidler <bazsi@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #5 from Balazs Scheidler <bazsi@balabit.hu> 2010-12-04 13:53:59 --- I've fixed this bug in all three supported branches of syslog-ng: 3.0, 3.1 and 3.2 (earlier versions were not affected anyway). This bug has been introduced in 3.0 and went unnoticed (or at least unreported) for several years. Thank you for finding and reporting it. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla@bugzilla.balabit.com