If it helps, some times the flush_mode is LW_FLUSH_NORMAL and other times LW_FLUSH_BUFFER #0 0x00007f4020000528 in ?? () #1 0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495 #2 0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043 #3 0x00007f402a1177ed in log_writer_work_perform (s=0xa0d460) at logwriter.c:129 #4 0x00007f402a117edb in main_loop_io_worker_job_start (self=0xa0d650) at mainloop.c:371 #5 0x00007f402a1341ca in iv_work_thread_do_work (_thr=0xabdd20) at iv_work.c:118 #6 0x00007f402a13352a in iv_run_tasks (st=0x7f400c40a130) at iv_task.c:48 #7 0x00007f402a13574c in iv_main () at iv_main_posix.c:106 #8 0x00007f402a133fe1 in iv_work_thread (_thr=0xabdd20) at iv_work.c:200 #9 0x00007f402a1361b8 in iv_thread_handler (_thr=0xabddf0) at iv_thread_posix.c:142 #10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f4028ad011d in clone () from /lib64/libc.so.6 On 02/22/2013 08:18 AM, Evan Rempel wrote:
On 02/08/2013 03:52 AM, Gergely Nagy wrote:
Evan Rempel <erempel@uvic.ca> writes:
We are seeing a different problem though. (happens on both 3.3.7 and 3.4.1 threaded and not threaded)
2013-02-06T23:59:05-08:00 kern.info kernel: syslog-ng[10913]: segfault at 7f819c000168 ip 00007f819c000168 sp 00007f81b33f5a48 error 15 2013-02-06T23:59:05-08:00 syslog.notice syslog-ng[7627]: Syslog connection closed; fd='13', client='AF_INET(142.104.47.145:51679)', local='AF_INET(127.0.0.1:1514)' 2013-02-06T23:59:05-08:00 daemon.crit supervise/syslog-ng[18771]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11' [...] So there does not seem to be a memory leak, but obviously something goes wrong to get a segfault.
I can't trace this for 4-5 days, so how do we trouble shoot this?
If you could enable core dumps, and get a backtrace, that would help a lot to narrow down the issue.
OK, I was able to capture a core dump and the backtrace looks like,
% sudo gdb /usr/local/sbin/syslog-ng core.11481
Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done. [New Thread 11954] [New Thread 12168] [New Thread 12185] [New Thread 12184] [New Thread 12060] [New Thread 11481] ... Core was generated by `/usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.server.c'. Program terminated with signal 11, Segmentation fault. #0 0x00007f40080008b0 in ?? () (gdb) backtrace #0 0x00007f40080008b0 in ?? () #1 0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495 #2 0x00007f402a11760c in log_writer_flush (self=0x9b3970, flush_mode=LW_FLUSH_BUFFER) at logwriter.c:1043 #3 0x00007f402a1177ed in log_writer_work_perform (s=0x9b3970) at logwriter.c:129 #4 0x00007f402a117edb in main_loop_io_worker_job_start (self=0x9b3b60) at mainloop.c:371 #5 0x00007f402a1341ca in iv_work_thread_do_work (_thr=0x70aa30) at iv_work.c:118 #6 0x00007f402a13352a in iv_run_tasks (st=0x7f40180dd200) at iv_task.c:48 #7 0x00007f402a13574c in iv_main () at iv_main_posix.c:106 #8 0x00007f402a133fe1 in iv_work_thread (_thr=0x70aa30) at iv_work.c:200 #9 0x00007f402a1361b8 in iv_thread_handler (_thr=0x940ae0) at iv_thread_posix.c:142 #10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f4028ad011d in clone () from /lib64/libc.so.6 (gdb)
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria