[Bug 183] New: Threaded Syslog-ng chashes resolving locally
https://bugzilla.balabit.com/show_bug.cgi?id=183 Summary: Threaded Syslog-ng chashes resolving locally Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: egon@local.ee Type of the Report: bug Estimated Hours: 0.0 Syslog-ng crashes with segfault with following combination: threaded(yes); use_dns(persist_only); dns_cache_hosts(/etc/hosts); Jun 8 11:32:09 kernel: [5626308.537129] syslog-ng[15155]: segfault at 10 ip 00007fb09a512208 sp 00007fb0962b52a0 error 4 in libc-2.13.so[7fb09a49a000+17d000] Also tried to compile from source and on Debian squeeze distro, same issue occurred. System info: Debian GNU/Linux wheezy/sid \n \l Linux dev1 2.6.32-5-xen-amd64 #1 SMP Mon Jan 16 20:48:30 UTC 2012 x86_64 dpkg -l | grep -e syslog-ng -e libevtlog ii libevtlog0 0.2.12-5 Syslog event logger library ii libsyslog-ng-3.3.5:amd64 3.3.5-2 Next generation system logging daemon (private library) ii syslog-ng 3.3.5-2 Next generation system logging daemon (metapackage) ii syslog-ng-core 3.3.5-2 Next generation system logging daemon (core) ii syslog-ng-mod-json 3.3.5-2 Next generation system logging daemon (JSON plugin) ii syslog-ng-mod-mongodb 3.3.5-2 Next generation system logging daemon (MongoDB plugin) ii syslog-ng-mod-sql 3.3.5-2 Next generation system logging daemon (SQL plugin) # /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid -F --enable-core *** glibc detected *** /usr/sbin/syslog-ng: double free or corruption (fasttop): 0x00007f2ef25584c0 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x75b46)[0x7f2eeef75b46] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f2eeef7a87c] /usr/lib/x86_64-linux-gnu/libsyslog-ng-3.3.5.so(main_loop_io_worker_thread_stop+0x65)[0x7f2ef0843755] /usr/lib/x86_64-linux-gnu/libsyslog-ng-3.3.5.so(+0x55b5b)[0x7f2ef085eb5b] /usr/lib/x86_64-linux-gnu/libsyslog-ng-3.3.5.so(+0x56226)[0x7f2ef085f226] /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f2eef28db50] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f2eeefd89dd] ======= Memory map: ======== 7f2ee4000000-7f2ee4022000 rw-p 00000000 00:00 0 7f2ee4022000-7f2ee8000000 ---p 00000000 00:00 0 7f2eea308000-7f2eea31d000 r-xp 00000000 ca:01 1180084 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f2eea31d000-7f2eea51d000 ---p 00015000 ca:01 1180084 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f2eea51d000-7f2eea51e000 rw-p 00015000 ca:01 1180084 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f2eea51e000-7f2eea51f000 ---p 00000000 00:00 0 7f2eea51f000-7f2eead1f000 rw-p 00000000 00:00 0 7f2eead1f000-7f2eead20000 ---p 00000000 00:00 0 7f2eead20000-7f2eeb520000 rw-p 00000000 00:00 0 7f2eeb520000-7f2eeb527000 r-xp 00000000 ca:01 2886205 /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1 7f2eeb527000-7f2eeb726000 ---p 00007000 ca:01 2886205 /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1 7f2eeb726000-7f2eeb727000 r--p 00006000 ca:01 2886205 /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1 7f2eeb727000-7f2eeb728000 rw-p 00007000 ca:01 2886205 /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1 7f2eeb728000-7f2eeb729000 r-xp 00000000 ca:01 3020751 /usr/lib/syslog-ng/3.3.5/libtfjson.so 7f2eeb729000-7f2eeb929000 ---p 00001000 ca:01 3020751 /usr/lib/syslog-ng/3.3.5/libtfjson.so 7f2eeb929000-7f2eeb92a000 r--p 00001000 ca:01 3020751 /usr/lib/syslog-ng/3.3.5/libtfjson.so 7f2eeb92a000-7f2eeb92b000 rw-p 00002000 ca:01 3020751 /usr/lib/syslog-ng/3.3.5/libtfjson.so 7f2eeb92b000-7f2eeb93d000 r-xp 00000000 ca:01 2886210 /usr/lib/x86_64-linux-gnu/libmongo-client.so.0.3.1 7f2eeb93d000-7f2eebb3d000 ---p 00012000 ca:01 2886210 /usr/lib/x86_64-linux-gnu/libmongo-client.so.0.3.1 7f2eebb3d000-7f2eebb3e000 r--p 00012000 ca:01 2886210 /usr/lib/x86_64-linux-gnu/libmongo-client.so.0.3.1 7f2eebb3e000-7f2eebb3f000 rw-p 00013000 ca:01 2886210 /usr/lib/x86_64-linux-gnu/libmongo-client.so.0.3.1 7f2eebb3f000-7f2eebb49000 r-xp 00000000 ca:01 3020746 /usr/lib/syslog-ng/3.3.5/libafmongodb.so 7f2eebb49000-7f2eebd48000 ---p 0000a000 ca:01 3020746 /usr/lib/syslog-ng/3.3.5/libafmongodb.so 7f2eebd48000-7f2eebd49000 r--p 00009000 ca:01 3020746 /usr/lib/syslog-ng/3.3.5/libafmongodb.so 7f2eebd49000-7f2eebd4a000 rw-p 0000a000 ca:01 3020746 /usr/lib/syslog-ng/3.3.5/libafmongodb.so 7f2eebd4a000-7f2eebd55000 r-xp 00000000 ca:01 1180101 /lib/x86_64-linux-gnu/libnss_files-2.13.so 7f2eebd55000-7f2eebf54000 ---p 0000b000 ca:01 1180101 /lib/x86_64-linux-gnu/libnss_files-2.13.so 7f2eebf54000-7f2eebf55000 r--p 0000a000 ca:01 1180101 /lib/x86_64-linux-gnu/libnss_files-2.13.so 7f2eebf55000-7f2eebf56000 rw-p 0000b000 ca:01 1180101 /lib/x86_64-linux-gnu/libnss_files-2.13.so 7f2eebf56000-7f2eebf60000 r-xp 00000000 ca:01 1180097 /lib/x86_64-linux-gnu/libnss_nis-2.13.so 7f2eebf60000-7f2eec15f000 ---p 0000a000 ca:01 1180097 /lib/x86_64-linux-gnu/libnss_nis-2.13.so 7f2eec15f000-7f2eec160000 r--p 00009000 ca:01 1180097 /lib/x86_64-linux-gnu/libnss_nis-2.13.so 7f2eec160000-7f2eec161000 rw-p 0000a000 ca:01 1180097 /lib/x86_64-linux-gnu/libnss_nis-2.13.so 7f2eec161000-7f2eec168000 r-xp 00000000 ca:01 1180090 /lib/x86_64-linux-gnu/libnss_compat-2.13.so 7f2eec168000-7f2eec367000 ---p 00007000 ca:01 1180090 /lib/x86_64-linux-gnu/libnss_compat-2.13.so 7f2eec367000-7f2eec368000 r--p 00006000 ca:01 1180090 /lib/x86_64-linux-gnu/libnss_compat-2.13.so 7f2eec368000-7f2eec369000 rw-p 00007000 ca:01 1180090 /lib/x86_64-linux-gnu/libnss_compat-2.13.so 7f2eec369000-7f2eec36b000 r-xp 00000000 ca:01 3015504 /usr/lib/syslog-ng/3.3.5/libconfgen.so 7f2eec36b000-7f2eec56a000 ---p 00002000 ca:01 3015504 /usr/lib/syslog-ng/3.3.5/libconfgen.so 7f2eec56a000-7f2eec56b000 r--p 00001000 ca:01 3015504 /usr/lib/syslog-ng/3.3.5/libconfgen.so 7f2eec56b000-7f2eec56c000 rw-p 00002000 ca:01 3015504 /usr/lib/syslog-ng/3.3.5/libconfgen.so 7f2eec56c000-7f2eec570000 r-xp 00000000 ca:01 3015523 /usr/lib/syslog-ng/3.3.5/libsyslogformat.so 7f2eec570000-7f2eec76f000 ---p 00004000 ca:01 3015523 /usr/lib/syslog-ng/3.3.5/libsyslogformat.so 7f2eec76f000-7f2eec770000 r--p 00003000 ca:01 3015523 /usr/lib/syslog-ng/3.3.5/libsyslogformat.so 7f2eec770000-7f2eec771000 rw-p 00004000 ca:01 3015523 /usr/lib/syslog-ng/3.3.5/libsyslogformat.so 7f2eec771000-7f2eec787000 r-xp 00000000 ca:01 3015515 /usr/lib/syslog-ng/3.3.5/libdbparser.so 7f2eec787000-7f2eec987000 ---p 00016000 ca:01 3015515 /usr/lib/syslog-ng/3.3.5/libdbparser.so 7f2eec987000-7f2eec988000 r--p 00016000 ca:01 3015515 /usr/lib/syslog-ng/3.3.5/libdbparser.so 7f2eec988000-7f2eec989000 rw-p 00017000 ca:01 3015515 /usr/lib/syslog-ng/3.3.5/libdbparser.so 7f2eec989000-7f2eec991000 r-xp 00000000 ca:01 3015509 /usr/lib/syslog-ng/3.3.5/libcsvparser.so 7f2eec991000-7f2eecb91000 ---p 00008000 ca:01 3015509 /usr/lib/syslog-ng/3.3.5/libcsvparser.so 7f2eecb91000-7f2eecb92000 r--p 00008000 ca:01 3015509 /usr/lib/syslog-ng/3.3.5/libcsvparser.so 7f2eecb92000-7f2eecb93000 rw-p 00009000 ca:01 3015509 /usr/lib/syslog-ng/3.3.5/libcsvparser.so 7f2eecb93000-7f2eecb95000 r-xp 00000000 ca:01 3015503 /usr/lib/syslog-ng/3.3.5/libbasicfuncs.so 7f2eecb95000-7f2eecd95000 ---p 00002000 ca:01 3015503 /usr/lib/syslog-ng/3.3.5/libbasicfuncs.so 7f2eecd95000-7f2eecd96000 r--p 00002000 ca:01 3015503 /usr/lib/syslog-ng/3.3.5/libbasicfuncs.so 7f2eecd96000-7f2eecd97000 rw-p 00003000 ca:01 3015503 /usr/lib/syslog-ng/3.3.5/libbasicfuncs.so 7f2eecd97000-7f2eece18000 r-xp 00000000 ca:01 1180092 /lib/x86_64-linux-gnu/libm-2.13.so 7f2eece18000-7f2eed017000 ---p 00081000 ca:01 1180092 /lib/x86_64-linux-gnu/libm-2.13.so 7f2eed017000-7f2eed018000 r--p 00080000 ca:01 1180092 /lib/x86_64-linux-gnu/libm-2.13.so 7f2eed018000-7f2eed019000 rw-p 00081000 ca:01 1180092 /lib/x86_64-linux-gnu/libm-2.13.soAborted (core dumped) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=183 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |3.3.6 CC| |algernon@balabit.hu AssignedTo|bazsi@balabit.hu |algernon@balabit.hu Severity|minor |major --- Comment #1 from Gergely Nagy <algernon@balabit.hu> 2012-06-08 15:08:27 --- If the problem surfaces again, can you send me the core file, please? I managed to reproduce it, but the backtrace looks weird: Thread 3 (Thread 0x7f73b783c700 (LWP 22411)): #0 0x00007f73bbab5f63 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f73bd351303 in iv_epoll_poll (numfds=1, active=0x7f73b78399d0, msec=10000) at ../../../../lib/ivykis/lib/iv_method_epoll.c:73 #2 0x00007f73bd350a6d in iv_main () at ../../../../lib/ivykis/lib/iv_main.c:265 #3 0x00007f73bd35552a in iv_work_thread (_thr=0x1fbbf20) at ../../../../lib/ivykis/modules/iv_work.c:196 #4 0x00007f73bd355b94 in iv_thread_handler (_thr=0x1fbbff0) at ../../../../lib/ivykis/modules/iv_thread.c:100 #5 0x00007f73bbd6ab50 in start_thread (arg=<optimized out>) at pthread_create.c:304 #6 0x00007f73bbab590d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 2 (Thread 0x7f73bd781b00 (LWP 22393)): #0 0x00007f73bbab5f63 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f73bd351303 in iv_epoll_poll (numfds=7, active=0x7fffab1013f0, msec=19683) at ../../../../lib/ivykis/lib/iv_method_epoll.c:73 #2 0x00007f73bd350a6d in iv_main () at ../../../../lib/ivykis/lib/iv_main.c:265 #3 0x00007f73bd32b0cf in main_loop_run () at ../../lib/mainloop.c:734 #4 0x0000000000401993 in main (argc=1, argv=0x7fffab101558) at ../../syslog-ng/main.c:260 Thread 1 (Thread 0x7f73b903f700 (LWP 22408)): #0 malloc_consolidate (av=0x7f73bbd5fe60) at malloc.c:5155 #1 0x00007f73bba543e8 in _int_free (av=0x7f73bbd5fe60, p=0x20102d0) at malloc.c:5034 #2 0x00007f73bba577ec in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738 #3 0x00007f73bc829c8f in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007f73bc829e7d in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f73bbd6a139 in __nptl_deallocate_tsd () at pthread_create.c:155 #6 0x00007f73bbd6ab5e in start_thread (arg=<optimized out>) at pthread_create.c:311 #7 0x00007f73bbab590d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #8 0x0000000000000000 in ?? () I wonder if you're seeing the same thing. I'll see if I can catch it with a debugger, or under valgrind. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=183 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=183 --- Comment #2 from Gergely Nagy <algernon@balabit.hu> 2012-06-08 15:17:53 --- (In reply to comment #1)
I managed to reproduce it, but the backtrace looks weird:
Right. Under gdb, it's much better: Thread 4 (Thread 0x7ffff288f700 (LWP 22936)): #0 0x00007ffff6261475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007ffff62646f0 in *__GI_abort () at abort.c:92 #2 0x00007ffff629b26b in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189 #3 0x00007ffff62a4ab6 in malloc_printerr (action=3, str=0x7ffff637b7e0 "double free or corruption (fasttop)", ptr=<optimized out>) at malloc.c:6283 #4 0x00007ffff62a97ec in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738 #5 0x00007ffff7b6202f in dns_cache_destroy () at ../../lib/dnscache.c:365 #6 0x00007ffff7b7c4ca in main_loop_io_worker_thread_stop (cookie=0x0) at ../../lib/mainloop.c:324 #7 0x00007ffff7ba754c in iv_work_thread (_thr=0x622bd0) at ../../../../lib/ivykis/modules/iv_work.c:199 #8 0x00007ffff7ba7b94 in iv_thread_handler (_thr=0x622ca0) at ../../../../lib/ivykis/modules/iv_thread.c:100 #9 0x00007ffff65bcb50 in start_thread (arg=<optimized out>) at pthread_create.c:304 #10 0x00007ffff630790d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #11 0x0000000000000000 in ?? () This seems to be the problem. I suppose dns_cache_destroy() is not guarded by a lock, while it should be. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=183 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|ASSIGNED |RESOLVED --- Comment #3 from Gergely Nagy <algernon@balabit.hu> 2012-06-08 15:40:29 --- Ok, for further reference, here's what happens: the main loop will init/destroy the DNS cache when worker threads start and stop, which would be fine, since all the dns cache variables are thread local. Except, dns_cache_destroy() also frees dns_cache_hosts, which is NOT thread local, but a global static. This results in memory corruption. Triggering the problem is trivial, too: set use_dns(persist_only) and dns_cache_hosts(/etc/hosts) along with threaded(yes) in the config, and dns_cache_expire(5) too. Send a log, disconnect, wait a few seconds for the worker thread to stop, connect again, and watch it blow up in a segfault. The solution is very simple: stop freeing dns_cache_host in dns_cache_destroy(), and introduce a dns_cache_deinit() function to do that, which is called when syslog-ng shuts down. I just applied a fix to the 3.3 branch[1] that does that, so this issue will be fixed in 3.3.6. Thanks for the report! [1]: http://git.madhouse-project.org/debian/syslog-ng/patch/?id=c0afec5b46eb3508f... -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla＠bugzilla.balabit.com