syslog-ng 3.3.2 seg faulting sending message via tcp
Syslog-ng 3.3.2 (obtained from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.2/source) is segfaulting when it tries to send a message to a TCP destination. I'm asking here instead of a bug report because I didnt see a 3.3.2 release announcement, so not sure if this is the final build or not. Anyway, gdb backtrace shows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x40f67940 (LWP 29440)] 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 288 s->queue(s, msg, path_options, s->queue_data); (gdb) where #0 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 #1 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at logmpx.c:125 #2 0x00002b047a80ce2c in log_pipe_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #3 0x00002b047a80cddb in log_pipe_forward_msg (self=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:275 #4 0x00002b047a80cd79 in log_filter_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at filter.c:698 #5 0x00002b047a812264 in log_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #6 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at logmpx.c:125 #7 0x00002b047a82cd20 in log_pipe_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #8 0x00002b047a82cccf in log_pipe_forward_msg (self=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #9 0x00002b047a82cc8a in log_source_group_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at sgroup.c:102 #10 0x00002b047a81ee1f in log_pipe_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #11 0x00002b047a81edce in log_pipe_forward_msg (self=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #12 0x00002b047a81ee32 in log_pipe_queue (s=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #13 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #14 0x00002b047a81ee32 in log_pipe_queue (s=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #15 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cd8800, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #16 0x00002b047a81eccc in log_source_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0, user_data=0x0) at logsource.c:290 #17 0x00002b047a81c969 in log_pipe_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0) at logpipe.h:288 #18 0x00002b047a81c919 in log_reader_handle_line (self=0x12cd8800, line=0x12d09800 "<22>Nov 14 02:48:00 postfix/smtpd[23425]: connect from slider.dev.usa.net[165.212.101.17]\n", length=89, saddr=0x0) at logreader.c:503 #19 0x00002b047a81cadd in log_reader_fetch_log (self=0x12cd8800) at logreader.c:561 #20 0x00002b047a81be25 in log_reader_work_perform (s=0x12cd8800) at logreader.c:115 #21 0x00002b047a824648 in main_loop_io_worker_job_start (self=0x12cd8a98) at mainloop.c:367 #22 0x00002b047a86ae96 in iv_work_thread_got_event (_thr=0x12cdb1f0) at iv_work.c:113 #23 0x00002b047a869e17 in iv_event_run_pending_events (_dummy=0x0) at iv_event.c:67 #24 0x00002b047a86a3db in iv_event_raw_got_event (_this=0x40f67858) at iv_event_raw.c:82 #25 0x00002b047a866945 in iv_run_active_list (active=0x40f64c20) at iv_main.c:219 #26 0x00002b047a866b83 in iv_main () at iv_main.c:269 #27 0x00002b047a86b290 in iv_work_thread (_thr=0x12cdb1f0) at iv_work.c:196 #28 0x00002b047a86b81a in iv_thread_handler (_thr=0x12cda400) at iv_thread.c:100 #29 0x00000039820064a7 in start_thread () from /lib64/libpthread.so.0 #30 0x00000039818d3c2d in clone () from /lib64/libc.so.6 Config: @version: 3.3 options { time_reopen(1); use_dns(no); use_fqdn(yes); keep_hostname(yes); normalize_hostnames(yes); create_dirs(yes); dir_perm(0755); perm(0644); mark_freq(0); threaded(yes); }; @include "scl.conf" source s_local { system(); internal(); }; filter f_mail { facility(mail); }; destination d_central { tcp('syslog.usa.net' port(514) log_fifo_size(50000)); }; log { source(s_local); filter(f_mail); destination(d_central); };
"Patrick H." <syslogng@feystorm.net> writes:
Anyway, gdb backtrace shows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x40f67940 (LWP 29440)] 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 288 s->queue(s, msg, path_options, s->queue_data); (gdb) where #0 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288
Oh dear, oh dear. I know this backtrace. I know it all too well. I'll see what happened, as this was supposed to be fixed in 3.3.2. -- |8]
On Sun, 2011-11-13 at 19:58 -0700, Patrick H. wrote:
Syslog-ng 3.3.2 (obtained from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.2/source) is segfaulting when it tries to send a message to a TCP destination. I'm asking here instead of a bug report because I didnt see a 3.3.2 release announcement, so not sure if this is the final build or not.
Anyway, gdb backtrace shows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x40f67940 (LWP 29440)] 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 288 s->queue(s, msg, path_options, s->queue_data); (gdb) where #0 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 #1 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at logmpx.c:125 #2 0x00002b047a80ce2c in log_pipe_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #3 0x00002b047a80cddb in log_pipe_forward_msg (self=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:275 #4 0x00002b047a80cd79 in log_filter_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at filter.c:698 #5 0x00002b047a812264 in log_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #6 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at logmpx.c:125 #7 0x00002b047a82cd20 in log_pipe_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #8 0x00002b047a82cccf in log_pipe_forward_msg (self=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #9 0x00002b047a82cc8a in log_source_group_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at sgroup.c:102 #10 0x00002b047a81ee1f in log_pipe_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #11 0x00002b047a81edce in log_pipe_forward_msg (self=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #12 0x00002b047a81ee32 in log_pipe_queue (s=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #13 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #14 0x00002b047a81ee32 in log_pipe_queue (s=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #15 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cd8800, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #16 0x00002b047a81eccc in log_source_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0, user_data=0x0) at logsource.c:290 #17 0x00002b047a81c969 in log_pipe_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0) at logpipe.h:288 #18 0x00002b047a81c919 in log_reader_handle_line (self=0x12cd8800, line=0x12d09800 "<22>Nov 14 02:48:00 postfix/smtpd[23425]: connect from slider.dev.usa.net[165.212.101.17]\n", length=89, saddr=0x0) at logreader.c:503 #19 0x00002b047a81cadd in log_reader_fetch_log (self=0x12cd8800) at logreader.c:561 #20 0x00002b047a81be25 in log_reader_work_perform (s=0x12cd8800) at logreader.c:115 #21 0x00002b047a824648 in main_loop_io_worker_job_start (self=0x12cd8a98) at mainloop.c:367 #22 0x00002b047a86ae96 in iv_work_thread_got_event (_thr=0x12cdb1f0) at iv_work.c:113 #23 0x00002b047a869e17 in iv_event_run_pending_events (_dummy=0x0) at iv_event.c:67 #24 0x00002b047a86a3db in iv_event_raw_got_event (_this=0x40f67858) at iv_event_raw.c:82 #25 0x00002b047a866945 in iv_run_active_list (active=0x40f64c20) at iv_main.c:219 #26 0x00002b047a866b83 in iv_main () at iv_main.c:269 #27 0x00002b047a86b290 in iv_work_thread (_thr=0x12cdb1f0) at iv_work.c:196 #28 0x00002b047a86b81a in iv_thread_handler (_thr=0x12cda400) at iv_thread.c:100 #29 0x00000039820064a7 in start_thread () from /lib64/libpthread.so.0 #30 0x00000039818d3c2d in clone () from /lib64/libc.so.6
bad news. it is the final build. :( and I just announced that to freshmat. -- Bazsi
On Sun, 2011-11-13 at 19:58 -0700, Patrick H. wrote:
Syslog-ng 3.3.2 (obtained from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.2/source) is segfaulting when it tries to send a message to a TCP destination. I'm asking here instead of a bug report because I didnt see a 3.3.2 release announcement, so not sure if this is the final build or not.
Anyway, gdb backtrace shows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x40f67940 (LWP 29440)] 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 288 s->queue(s, msg, path_options, s->queue_data); (gdb) where #0 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 #1 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at logmpx.c:125 #2 0x00002b047a80ce2c in log_pipe_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #3 0x00002b047a80cddb in log_pipe_forward_msg (self=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:275 #4 0x00002b047a80cd79 in log_filter_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at filter.c:698 #5 0x00002b047a812264 in log_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #6 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at logmpx.c:125 #7 0x00002b047a82cd20 in log_pipe_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #8 0x00002b047a82cccf in log_pipe_forward_msg (self=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #9 0x00002b047a82cc8a in log_source_group_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at sgroup.c:102 #10 0x00002b047a81ee1f in log_pipe_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #11 0x00002b047a81edce in log_pipe_forward_msg (self=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #12 0x00002b047a81ee32 in log_pipe_queue (s=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #13 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #14 0x00002b047a81ee32 in log_pipe_queue (s=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #15 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cd8800, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #16 0x00002b047a81eccc in log_source_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0, user_data=0x0) at logsource.c:290 #17 0x00002b047a81c969 in log_pipe_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0) at logpipe.h:288 #18 0x00002b047a81c919 in log_reader_handle_line (self=0x12cd8800, line=0x12d09800 "<22>Nov 14 02:48:00 postfix/smtpd[23425]: connect from slider.dev.usa.net[165.212.101.17]\n", length=89, saddr=0x0) at logreader.c:503 #19 0x00002b047a81cadd in log_reader_fetch_log (self=0x12cd8800) at logreader.c:561 #20 0x00002b047a81be25 in log_reader_work_perform (s=0x12cd8800) at logreader.c:115 #21 0x00002b047a824648 in main_loop_io_worker_job_start (self=0x12cd8a98) at mainloop.c:367 #22 0x00002b047a86ae96 in iv_work_thread_got_event (_thr=0x12cdb1f0) at iv_work.c:113 #23 0x00002b047a869e17 in iv_event_run_pending_events (_dummy=0x0) at iv_event.c:67 #24 0x00002b047a86a3db in iv_event_raw_got_event (_this=0x40f67858) at iv_event_raw.c:82 #25 0x00002b047a866945 in iv_run_active_list (active=0x40f64c20) at iv_main.c:219 #26 0x00002b047a866b83 in iv_main () at iv_main.c:269 #27 0x00002b047a86b290 in iv_work_thread (_thr=0x12cdb1f0) at iv_work.c:196 #28 0x00002b047a86b81a in iv_thread_handler (_thr=0x12cda400) at iv_thread.c:100 #29 0x00000039820064a7 in start_thread () from /lib64/libpthread.so.0 #30 0x00000039818d3c2d in clone () from /lib64/libc.so.6
I could reproduce a crash here, and this one solves it. (was related to one of the leak fixes which went in before 3.3.2 was released) $ git diff diff --git a/lib/driver.h b/lib/driver.h index c805da5..7f7acbd 100644 --- a/lib/driver.h +++ b/lib/driver.h @@ -188,7 +188,6 @@ log_dest_driver_release_queue(LogDestDriver *self, LogQueue *q) if (q) { self->queues = g_list_remove(self->queues, q); - log_queue_unref(q); self->release_queue(self, q, self->release_queue_data); } I still have to think through all consequences whether this reintroduces a leak, but this seems to be right, I just have to go now. PS: it warrants a 3.3.2 if this solves the issue for you. -- Bazsi
Sent: Tue Nov 15 2011 12:37:46 GMT-0700 (MST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng 3.3.2 seg faulting sending message via tcp
On Sun, 2011-11-13 at 19:58 -0700, Patrick H. wrote:
Syslog-ng 3.3.2 (obtained from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.2/source) is segfaulting when it tries to send a message to a TCP destination. I'm asking here instead of a bug report because I didnt see a 3.3.2 release announcement, so not sure if this is the final build or not.
Anyway, gdb backtrace shows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x40f67940 (LWP 29440)] 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 288 s->queue(s, msg, path_options, s->queue_data); ... I could reproduce a crash here, and this one solves it. (was related to one of the leak fixes which went in before 3.3.2 was released)
$ git diff diff --git a/lib/driver.h b/lib/driver.h index c805da5..7f7acbd 100644 --- a/lib/driver.h +++ b/lib/driver.h @@ -188,7 +188,6 @@ log_dest_driver_release_queue(LogDestDriver *self, LogQueue *q) if (q) { self->queues = g_list_remove(self->queues, q); - log_queue_unref(q);
self->release_queue(self, q, self->release_queue_data); }
I still have to think through all consequences whether this reintroduces a leak, but this seems to be right, I just have to go now.
PS: it warrants a 3.3.2 if this solves the issue for you.
I've been running this on several machines through the day and they all appear stable. Thanks
----- Original message -----
Sent: Tue Nov 15 2011 12:37:46 GMT-0700 (MST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng 3.3.2 seg faulting sending message via tcp
On Sun, 2011-11-13 at 19:58 -0700, Patrick H. wrote:
Syslog-ng 3.3.2 (obtained from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.2/source) is segfaulting when it tries to send a message to a TCP destination. I'm asking here instead of a bug report because I didnt see a 3.3.2 release announcement, so not sure if this is the final build or not.
Anyway, gdb backtrace shows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x40f67940 (LWP 29440)] 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 288 s->queue(s, msg, path_options, s->queue_data); ... I could reproduce a crash here, and this one solves it. (was related to one of the leak fixes which went in before 3.3.2 was released)
$ git diff diff --git a/lib/driver.h b/lib/driver.h index c805da5..7f7acbd 100644 --- a/lib/driver.h +++ b/lib/driver.h @@ -188,7 +188,6 @@ log_dest_driver_release_queue(LogDestDriver *self, LogQueue *q) if (q) { self->queues = g_list_remove(self->queues, q); - log_queue_unref(q);
self->release_queue(self, q, self->release_queue_data); }
I still have to think through all consequences whether this reintroduces a leak, but this seems to be right, I just have to go now.
PS: it warrants a 3.3.2 if this solves the issue for you.
I've been running this on several machines through the day and they all appear stable.
great news. I'll do a quick validation and release 3.3.3 before the masses get hit by this problem. thanks for the feedback.
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
Patrick H.