[syslog-ng] segfault - backtrace included

[Evan Rempel]

If it helps, some times the flush_mode is LW_FLUSH_NORMAL and other times LW_FLUSH_BUFFER

#0  0x00007f4020000528 in ?? ()
#1  0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495
#2  0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043
#3  0x00007f402a1177ed in log_writer_work_perform (s=0xa0d460) at logwriter.c:129
#4  0x00007f402a117edb in main_loop_io_worker_job_start (self=0xa0d650) at mainloop.c:371
#5  0x00007f402a1341ca in iv_work_thread_do_work (_thr=0xabdd20) at iv_work.c:118
#6  0x00007f402a13352a in iv_run_tasks (st=0x7f400c40a130) at iv_task.c:48
#7  0x00007f402a13574c in iv_main () at iv_main_posix.c:106
#8  0x00007f402a133fe1 in iv_work_thread (_thr=0xabdd20) at iv_work.c:200
#9  0x00007f402a1361b8 in iv_thread_handler (_thr=0xabddf0) at iv_thread_posix.c:142
#10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f4028ad011d in clone () from /lib64/libc.so.6


On 02/22/2013 08:18 AM, Evan Rempel wrote:
> On 02/08/2013 03:52 AM, Gergely Nagy wrote:
>> Evan Rempel <erempel at uvic.ca> writes:
>>
>>> We are seeing a different problem though. (happens on both 3.3.7 and 3.4.1 threaded and not threaded)
>>>
>>> 2013-02-06T23:59:05-08:00 kern.info kernel: syslog-ng[10913]: segfault at 7f819c000168 ip 00007f819c000168 sp 00007f81b33f5a48 error 15
>>> 2013-02-06T23:59:05-08:00 syslog.notice syslog-ng[7627]: Syslog connection closed; fd='13', client='AF_INET(142.104.47.145:51679)', local='AF_INET(127.0.0.1:1514)'
>>> 2013-02-06T23:59:05-08:00 daemon.crit supervise/syslog-ng[18771]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11'
>> [...]
>>> So there does not seem to be a memory leak, but obviously something goes wrong to get a segfault.
>>>
>>> I can't trace this for 4-5 days, so how do we trouble shoot this?
>>
>> If you could enable core dumps, and get a backtrace, that would help a
>> lot to narrow down the issue.
>>
>
> OK, I was able to capture a core dump and the backtrace looks like,
>
> % sudo gdb /usr/local/sbin/syslog-ng core.11481
>
> Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done.
> [New Thread 11954]
> [New Thread 12168]
> [New Thread 12185]
> [New Thread 12184]
> [New Thread 12060]
> [New Thread 11481]
> ...
> Core was generated by `/usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.server.c'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f40080008b0 in ?? ()
> (gdb) backtrace
> #0  0x00007f40080008b0 in ?? ()
> #1  0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495
> #2  0x00007f402a11760c in log_writer_flush (self=0x9b3970, flush_mode=LW_FLUSH_BUFFER) at logwriter.c:1043
> #3  0x00007f402a1177ed in log_writer_work_perform (s=0x9b3970) at logwriter.c:129
> #4  0x00007f402a117edb in main_loop_io_worker_job_start (self=0x9b3b60) at mainloop.c:371
> #5  0x00007f402a1341ca in iv_work_thread_do_work (_thr=0x70aa30) at iv_work.c:118
> #6  0x00007f402a13352a in iv_run_tasks (st=0x7f40180dd200) at iv_task.c:48
> #7  0x00007f402a13574c in iv_main () at iv_main_posix.c:106
> #8  0x00007f402a133fe1 in iv_work_thread (_thr=0x70aa30) at iv_work.c:200
> #9  0x00007f402a1361b8 in iv_thread_handler (_thr=0x940ae0) at iv_thread_posix.c:142
> #10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0
> #11 0x00007f4028ad011d in clone () from /lib64/libc.so.6
> (gdb)
>
>


-- 
Evan Rempel                                      erempel at uvic.ca
Senior Systems Administrator                        250.721.7691
Data Centre Services, University Systems, University of Victoria