[syslog-ng] [Bug 161] New: Syslog-ng stops processing after reload

bugzilla at bugzilla.balabit.com bugzilla at bugzilla.balabit.com
Tue Feb 14 15:50:51 CET 2012 

https://bugzilla.balabit.com/show_bug.cgi?id=161

           Summary: Syslog-ng stops processing after reload
           Product: syslog-ng
           Version: 3.3.x
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: unspecified
         Component: syslog-ng
        AssignedTo: bazsi at balabit.hu
        ReportedBy: bugzilla.syslogng at feystorm.net
Type of the Report: ---
   Estimated Hours: 0.0


syslog-ng version: 3.3.4
OS: RHEL 6.2

Whenever I issue a reload to our syslog-ng instance syslog-ng stops processing messages and sits there doing nothing.
The issue only seems to occur when the server is processing a high log volume. This server is part of a 2-node active/passive HA group. So we have 2 servers
with the exact same configuration. Whenever I issue the reload on the server that is not currently active, it works fine. I can issue a failover and the server
that was able to reload fine now hangs. The active server in this HA group processes around 35,000 messages per second.

I havent posted the config because it is fairly large, however I can if desired. The most notable thing in it is that threading is enabled.

Below is the backtrace of syslog-ng stuck in this state. I took a gcore, so I can gather any additional information that is needed.
Also, if I try to do a full backtrace ('where full') in gdb on any thread other than thread 5, gdb segfaults. Not sure if that has any relevance or not.



(gdb) info threads
  5 Thread 0x7fb9c654ab00 (LWP 11040)  0x000000353cedd967 in writev () from /lib64/libc.so.6
  4 Thread 0x7fb9c25fb700 (LWP 11049)  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
  3 Thread 0x7fb9c2ffc700 (LWP 11048)  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
  2 Thread 0x7fb9c39fd700 (LWP 11046)  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
* 1 Thread 0x7fb9c43fe700 (LWP 11045)  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6

(gdb) where
#0  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c43fb9a0, msec=10000) at iv_method_epoll.c:73
#2  0x00007fb9c65cf1bb in iv_main () at iv_main.c:265
#3  0x00007fb9c65d3c36 in iv_work_thread (_thr=0x14fc610) at iv_work.c:196
#4  0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc6e0) at iv_thread.c:100
#5  0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0
#6  0x000000353cee570d in clone () from /lib64/libc.so.6

(gdb) thread 2
[Switching to thread 2 (Thread 0x7fb9c39fd700 (LWP 11046))]#0  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
(gdb) where
#0  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c39fa9a0, msec=10000) at iv_method_epoll.c:73
#2  0x00007fb9c65cf1bb in iv_main () at iv_main.c:265
#3  0x00007fb9c65d3c36 in iv_work_thread (_thr=0x14fc8e0) at iv_work.c:196
#4  0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc9b0) at iv_thread.c:100
#5  0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0
#6  0x000000353cee570d in clone () from /lib64/libc.so.6

(gdb) thread 3
[Switching to thread 3 (Thread 0x7fb9c2ffc700 (LWP 11048))]#0  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
(gdb) where
#0  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c2ff99a0, msec=10000) at iv_method_epoll.c:73
#2  0x00007fb9c65cf1bb in iv_main () at iv_main.c:265
#3  0x00007fb9c65d3c36 in iv_work_thread (_thr=0x15635c0) at iv_work.c:196
#4  0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc380) at iv_thread.c:100
#5  0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0
#6  0x000000353cee570d in clone () from /lib64/libc.so.6

(gdb) thread 4
[Switching to thread 4 (Thread 0x7fb9c25fb700 (LWP 11049))]#0  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
(gdb) where
#0  0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c25f89a0, msec=10000) at iv_method_epoll.c:73
#2  0x00007fb9c65cf1bb in iv_main () at iv_main.c:265
#3  0x00007fb9c65d3c36 in iv_work_thread (_thr=0x1566d20) at iv_work.c:196
#4  0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x1566a70) at iv_thread.c:100
#5  0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0
#6  0x000000353cee570d in clone () from /lib64/libc.so.6

(gdb) thread 5
[Switching to thread 5 (Thread 0x7fb9c654ab00 (LWP 11040))]#0  0x000000353cedd967 in writev () from /lib64/libc.so.6
(gdb) where
#0  0x000000353cedd967 in writev () from /lib64/libc.so.6
#1  0x00007fb9c65820f8 in log_proto_file_writer_flush (s=0x171b5a0) at logproto.c:270
#2  0x00007fb9c658c403 in log_proto_flush (s=0x171b5a0) at logproto.h:86
#3  0x00007fb9c658ec3e in log_writer_flush (self=0x171adb0, flush_mode=LW_FLUSH_QUEUE) at logwriter.c:1032
#4  0x00007fb9c658f006 in log_writer_deinit (s=0x171adb0) at logwriter.c:1106
#5  0x00007fb9c63399d6 in log_pipe_deinit (s=0x171adb0) at ../../lib/logpipe.h:254
#6  0x00007fb9c633b3cd in affile_dw_deinit (s=0x171ac20) at affile.c:627
#7  0x00007fb9c63399d6 in log_pipe_deinit (s=0x171ac20) at ../../lib/logpipe.h:254
#8  0x00007fb9c633c211 in affile_dd_deinit_writer (key=0x171ad60, value=0x171ac20, user_data=0x0) at affile.c:974
#9  0x000000353e22a7a3 in g_hash_table_foreach () from /lib64/libglib-2.0.so.0
#10 0x00007fb9c633c349 in affile_dd_deinit (s=0x1539990) at affile.c:996
#11 0x00007fb9c6575d86 in log_pipe_deinit (s=0x1539990) at logpipe.h:254
#12 0x00007fb9c65761b4 in log_dest_group_deinit (s=0x1539ef0) at dgroup.c:76
#13 0x00007fb9c656f54c in log_pipe_deinit (s=0x1539ef0) at logpipe.h:254
#14 0x00007fb9c657078c in log_center_deinit (self=0x14f35c0) at center.c:576
#15 0x00007fb9c6571296 in cfg_deinit (cfg=0x14f1ca0) at cfg.c:261
#16 0x00007fb9c6590cac in main_loop_reload_config_apply () at mainloop.c:494
#17 0x00007fb9c6590aab in main_loop_io_worker_job_complete (self=0x156d160) at mainloop.c:417
#18 0x00007fb9c65d3ccc in iv_work_event (_pool=0x14f1bf0) at iv_work.c:221
#19 0x00007fb9c65d2862 in iv_event_run_pending_events (_dummy=0x0) at iv_event.c:67
#20 0x00007fb9c65d2cb5 in iv_event_raw_got_event (_this=0x7fb9c654aa18) at iv_event_raw.c:82
#21 0x00007fb9c65cefec in iv_run_active_list (active=0x7fff33ed17b0) at iv_main.c:219
#22 0x00007fb9c65cf1cc in iv_main () at iv_main.c:269
#23 0x00007fb9c65913e9 in main_loop_run () at mainloop.c:731
#24 0x00000000004016d0 in main ()


-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.

More information about the syslog-ng mailing list