[syslog-ng] segfault - backtrace included

[Evan Rempel]

On 02/08/2013 03:52 AM, Gergely Nagy wrote:
> Evan Rempel <erempel at uvic.ca> writes:
>
>> We are seeing a different problem though. (happens on both 3.3.7 and 3.4.1 threaded and not threaded)
>>
>> 2013-02-06T23:59:05-08:00 kern.info kernel: syslog-ng[10913]: segfault at 7f819c000168 ip 00007f819c000168 sp 00007f81b33f5a48 error 15
>> 2013-02-06T23:59:05-08:00 syslog.notice syslog-ng[7627]: Syslog connection closed; fd='13', client='AF_INET(142.104.47.145:51679)', local='AF_INET(127.0.0.1:1514)'
>> 2013-02-06T23:59:05-08:00 daemon.crit supervise/syslog-ng[18771]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11'
> [...]
>> So there does not seem to be a memory leak, but obviously something goes wrong to get a segfault.
>>
>> I can't trace this for 4-5 days, so how do we trouble shoot this?
>
> If you could enable core dumps, and get a backtrace, that would help a
> lot to narrow down the issue.
>

OK, I was able to capture a core dump and the backtrace looks like,

% sudo gdb /usr/local/sbin/syslog-ng core.11481

Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done.
[New Thread 11954]
[New Thread 12168]
[New Thread 12185]
[New Thread 12184]
[New Thread 12060]
[New Thread 11481]
...
Core was generated by `/usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.server.c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f40080008b0 in ?? ()
(gdb) backtrace
#0  0x00007f40080008b0 in ?? ()
#1  0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495
#2  0x00007f402a11760c in log_writer_flush (self=0x9b3970, flush_mode=LW_FLUSH_BUFFER) at logwriter.c:1043
#3  0x00007f402a1177ed in log_writer_work_perform (s=0x9b3970) at logwriter.c:129
#4  0x00007f402a117edb in main_loop_io_worker_job_start (self=0x9b3b60) at mainloop.c:371
#5  0x00007f402a1341ca in iv_work_thread_do_work (_thr=0x70aa30) at iv_work.c:118
#6  0x00007f402a13352a in iv_run_tasks (st=0x7f40180dd200) at iv_task.c:48
#7  0x00007f402a13574c in iv_main () at iv_main_posix.c:106
#8  0x00007f402a133fe1 in iv_work_thread (_thr=0x70aa30) at iv_work.c:200
#9  0x00007f402a1361b8 in iv_thread_handler (_thr=0x940ae0) at iv_thread_posix.c:142
#10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f4028ad011d in clone () from /lib64/libc.so.6
(gdb)


-- 
Evan Rempel                                      erempel at uvic.ca
Senior Systems Administrator                        250.721.7691
Data Centre Services, University Systems, University of Victoria