Hello, I tried to compile and run alpha1 on openSUSE 12.1 and FreeBSD 8.1. I got a segfault on start on both. Here are the backtraces: fb81# gdb74 -c syslog-ng.core /usr/local/sbin/syslog-ng GNU gdb (GDB) 7.4 [GDB v7.4 for FreeBSD] Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd8.1". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done. [New process 100055] [New Thread 8018041c0 (LWP 100055)] Core was generated by `syslog-ng'. Program terminated with signal 11, Segmentation fault. #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 60 file-perms.c: No such file or directory. (gdb) backtrace #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 #1 0x0000000801c0cda0 in afsocket_parse (lexer=0x80196b000, instance=0x7fffffffc788, arg=0x0) at afsocket-grammar.y:1186 #2 0x000000080068b4f3 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=0x80196b000, self=0x801d16fc0) at cfg-parser.h:83 #3 plugin_parse_config (self=0x801d174b0, cfg=<optimized out>, yylloc=<optimized out>, arg=0x0) at plugin.c:211 #4 0x0000000800698730 in main_parse (lexer=0x80196b000, dummy=0x7fffffffec48, arg=0x0) at cfg-grammar.y:518 #5 0x0000000800669ef9 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=<optimized out>, self=<optimized out>) at cfg-parser.h:83 #6 cfg_run_parser (self=0x801949180, lexer=0x80196b000, parser=0x8007c4140, result=0x7fffffffec48, arg=0x801c0f760) at cfg.c:316 #7 0x000000080066a006 in cfg_read_config (self=0x801949180, fname=0x8006ab16b "/usr/local/etc/syslog-ng.conf", syntax_only=<optimized out>, preprocess_into=0x0) at cfg.c:347 #8 0x0000000800685baa in main_loop_init () at mainloop.c:674 #9 0x000000000040171f in main () (gdb) And the same core file with gdb from the base system: fb81# gdb -c syslog-ng.core /usr/local/sbin/syslog-ng GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Core was generated by `syslog-ng'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/lib/libsyslog-ng-3.4.0alpha1.so...done. Loaded symbols for /usr/local/lib/libsyslog-ng-3.4.0alpha1.so Reading symbols from /usr/local/lib/libnet115/libnet.so.7...done. Loaded symbols for /usr/local/lib/libnet115/libnet.so.7 Reading symbols from /usr/lib/librt.so.1...done. Loaded symbols for /usr/lib/librt.so.1 Reading symbols from /usr/local/lib/libgmodule-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgmodule-2.0.so.0 Reading symbols from /usr/local/lib/libgthread-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgthread-2.0.so.0 Reading symbols from /usr/local/lib/libglib-2.0.so.0...done. Loaded symbols for /usr/local/lib/libglib-2.0.so.0 Reading symbols from /usr/local/lib/libintl.so.9...done. Loaded symbols for /usr/local/lib/libintl.so.9 Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /usr/local/lib/libevtlog.so.0...done. Loaded symbols for /usr/local/lib/libevtlog.so.0 Reading symbols from /usr/local/lib/libpcre.so.1...done. Loaded symbols for /usr/local/lib/libpcre.so.1 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/syslog-ng/libconfgen.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libconfgen.so Reading symbols from /usr/lib/libz.so...done. Loaded symbols for /usr/lib/libz.so Reading symbols from /usr/local/lib/syslog-ng/libafsocket.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libafsocket.so Reading symbols from /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so Reading symbols from /usr/local/lib/libcrypto.so.7...done. Loaded symbols for /usr/local/lib/libcrypto.so.7 Reading symbols from /usr/local/lib/libssl.so.7...done. Loaded symbols for /usr/local/lib/libssl.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 60 file-perms.c: No such file or directory. in file-perms.c [New Thread 8018041c0 (LWP 100055)] (gdb) backtrace #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 #1 0x0000000801c0cda0 in afsocket_parse (lexer=0x80196b000, instance=0x7fffffffc788, arg=0x0) at afsocket-grammar.y:1186 #2 0x000000080068b4f3 in plugin_parse_config (self=0x801d174b0, cfg=Variable "cfg" is not available. ) at cfg-parser.h:83 #3 0x0000000800698730 in main_parse (lexer=0x80196b000, dummy=0x7fffffffec48, arg=0x0) at cfg-grammar.y:518 #4 0x0000000800669ef9 in cfg_run_parser (self=0x801949180, lexer=0x80196b000, parser=0x8007c4140, result=0x7fffffffec48, arg=Variable "arg" is not available. ) at cfg-parser.h:83 #5 0x000000080066a006 in cfg_read_config (self=0x801949180, fname=0x8006ab16b "/usr/local/etc/syslog-ng.conf", syntax_only=Variable "syntax_only" is not available. ) at cfg.c:347 #6 0x0000000800685baa in main_loop_init () at mainloop.c:674 #7 0x000000000040171f in main () (gdb) This binary was compiled by copying sysutils/syslog-ng port to syslog-ng-devel, adding --enable-debug to configure args. I ran into one more interesting thing: with json-c enabled in "make config", configure failed with: configure: error: Cannot find json-c version >= 0.9: is pkg-config in path? When disabled json-c in "make config" it was found by configure and was built. The relevant part of "Makefile" is: .if defined(WITH_JSON_GLIB) LIB_DEPENDS+= json-glib-1.0.0:${PORTSDIR}/devel/json-glib CONFIGURE_ARGS+= --enable-json --with-json=json-glib PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .elif defined(WITH_JSON_C) LIB_DEPENDS+= json.0:${PORTSDIR}/devel/json-c CONFIGURE_ARGS+= --enable-json --with-json=json-c PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .else CONFIGURE_ARGS+= --disable-json .endif So, in theory json support was disabled... And another backtrace, this time from openSUSE: (gdb) backtrace #0 0xb7684e8d in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #1 0xb7684f60 in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #2 0xb7685557 in cfg_tree_compile_rule () from /lib/libsyslog-ng-3.4.0alpha1.so #3 0xb7685954 in cfg_tree_compile () from /lib/libsyslog-ng-3.4.0alpha1.so #4 0xb76859ee in cfg_tree_start () from /lib/libsyslog-ng-3.4.0alpha1.so #5 0xb76808c0 in cfg_init () from /lib/libsyslog-ng-3.4.0alpha1.so #6 0xb76a07ea in main_loop_init () from /lib/libsyslog-ng-3.4.0alpha1.so #7 0x080491a7 in main () (gdb) Bye, -- Peter Czanik (CzP)<czanik@balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/
Peter Czanik <czanik@balabit.hu> writes:
Hello,
I tried to compile and run alpha1 on openSUSE 12.1 and FreeBSD 8.1. I got a segfault on start on both. Here are the backtraces:
A little more info about this: $ sbin/syslog-ng syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' WARNING: Configuration file format is too old, please update it to use the 3.4 format as some constructs might operate inefficiently; *** glibc detected *** sbin/syslog-ng: double free or corruption (fasttop): 0x00000000011508e0 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x75ab6)[0x7f2c4318dab6] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f2c431927ec] /home/algernon/install/sng/3.4/lib/syslog-ng/libafmongodb.so(afmongodb_parse+0xf4e)[0x7f2c4021765d] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(+0x4caa8)[0x7f2c44a6eaa8] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(plugin_parse_config+0x174)[0x7f2c44a6f201] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(main_parse+0x108a)[0x7f2c44a83aba] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(+0x23528)[0x7f2c44a45528] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(cfg_run_parser+0x124)[0x7f2c44a46171] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(cfg_read_config+0x84)[0x7f2c44a4628d] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(main_loop_init+0xa9)[0x7f2c44a68ed9] sbin/syslog-ng(main+0x1cf)[0x401994] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f2c43136ead] sbin/syslog-ng[0x401549] Working from here: #5 0x00007ffff332665d in afmongodb_parse (lexer=0x61d6f0, instance=0x7fffffffa870, arg=0x0) at afmongodb-grammar.y:799 799 | KW_REPLACE '(' string string ')' { value_pairs_transform_set_add_func(last_vp_transset, value_pairs_new_transform_replace($3, $4)); free($3); free($4); } I tried playing with GDB and breaking after value_pairs_transform_set_add_func(), to see where things get freed twice, but after a few minutes of not being able to make heads and tails out of the bison-generated code I gave up. But perhaps the above will shed some light on where the problem may be. The next step I'd take is try to find a version of 3.4 that works, and git bisect. It does look like the problem is somewhere in the grammar stuff, as both backtraces come from ${module}_parse, and it affects multiple modules. The interesting thing is that make check still works, which suggests there is something in our config that triggers the bug. Furthermore, if I remove the @module lines from my config, I get a proper segfault: Program received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff65a9e60) at malloc.c:5155 5155 malloc.c: No such file or directory. (gdb) bt #0 malloc_consolidate (av=0x7ffff65a9e60) at malloc.c:5155 #1 0x00007ffff629f214 in _int_malloc (av=0x7ffff65a9e60, bytes=1221) at malloc.c:4373 #2 0x00007ffff62a0eb2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:4065 #3 0x00007ffff7de780f in ?? () from /lib64/ld-linux-x86-64.so.2 #4 0x00007ffff7de2fb6 in ?? () from /lib64/ld-linux-x86-64.so.2 #5 0x00007ffff7de46b8 in ?? () from /lib64/ld-linux-x86-64.so.2 #6 0x00007ffff7deee5e in ?? () from /lib64/ld-linux-x86-64.so.2 #7 0x00007ffff7dea926 in ?? () from /lib64/ld-linux-x86-64.so.2 #8 0x00007ffff7dee89a in ?? () from /lib64/ld-linux-x86-64.so.2 #9 0x00007ffff67caf66 in dlopen_doit (a=<optimized out>) at dlopen.c:67 #10 0x00007ffff7dea926 in ?? () from /lib64/ld-linux-x86-64.so.2 #11 0x00007ffff67cb2ec in _dlerror_run (operate=0x7ffff67caf00 <dlopen_doit>, args=0x7fffffffa6a0) at dlerror.c:164 #12 0x00007ffff67caee1 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:88 #13 0x00007ffff7717944 in _g_module_open (bind_local=<optimized out>, bind_lazy=<optimized out>, file_name=0x6e1900 "/home/algernon/install/sng/3.4/lib/syslog-ng/libafmongodb.so") at /tmp/buildd/glib2.0-2.30.2/./gmodule/gmodule-dl.c:99 #14 g_module_open (file_name=0x618e10 "/home/algernon/install/sng/3.4/lib/syslog-ng/libafmongodb.so", flags=<optimized out>) at /tmp/buildd/glib2.0-2.30.2/./gmodule/gmodule.c:439 #15 0x00007ffff7b7e4d4 in plugin_dlopen_module (module_name=0x648280 "afmongodb", module_path=0x618a30 "/home/algernon/install/sng/3.4/lib/syslog-ng") at ../../lib/plugin.c:305 #16 0x00007ffff7b7e626 in plugin_load_module (module_name=0x648280 "afmongodb", cfg=0x617900, args=0x0) at ../../lib/plugin.c:349 #17 0x00007ffff7b7df56 in plugin_find (cfg=0x617900, plugin_type=2, plugin_name=0x6e0eb0 "mongodb") at ../../lib/plugin.c:148 #18 0x00007ffff7b92a1d in main_parse (lexer=0x61d6f0, dummy=0x7fffffffcd98, arg=0x0) at cfg-grammar.y:607 #19 0x00007ffff7b54528 in cfg_parser_parse (self=0x7ffff7dd9aa0, lexer=0x61d6f0, instance=0x7fffffffcd98, arg=0x0) at ../../lib/cfg-parser.h:83 #20 0x00007ffff7b55171 in cfg_run_parser (self=0x617900, lexer=0x61d6f0, parser=0x7ffff7dd9aa0, result=0x7fffffffcd98, arg=0x0) at ../../lib/cfg.c:316 #21 0x00007ffff7b5528d in cfg_read_config (self=0x617900, fname=0x7ffff7baa960 "/home/algernon/install/sng/3.4/etc/syslog-ng.conf", syntax_only=0, preprocess_into=0x0) at ../../lib/cfg.c:347 #22 0x00007ffff7b77ed9 in main_loop_init () at ../../lib/mainloop.c:674 #23 0x0000000000401994 in main (argc=1, argv=0x7fffffffcf08) at ../../syslog-ng/main.c:239 Earlier: Module loaded and initialized successfully; module='confgen' Finishing include; filename='/home/algernon/install/sng/3.4/share/include/scl/syslogconf/plugin.conf', depth='2' Finishing include; filename='/home/algernon/install/sng/3.4/etc/scl.conf', depth='1' Module loaded and initialized successfully; module='afsocket-tls' Module loaded and initialized successfully; module='jsonparser' Module loaded and initialized successfully; module='affile' Module loaded and initialized successfully; module='tfjson' Using a simpler config like this: @version: 3.4 @include "scl.conf" options { threaded(yes); }; source s_local { # system(); internal(); }; destination d_local { file("/tmp/test.json" template("$(format-json --key json.* --rekey --shift 4 --add-prefix JSON --replace JSON.foo=foo --key cee.*)\n")); }; log { source(s_local); destination(d_local); }; I get the following backtrace: (gdb) bt #0 malloc_consolidate (av=0x7ffff65a9e60) at malloc.c:5155 #1 0x00007ffff629e3e8 in _int_free (av=0x7ffff65a9e60, p=0x621740) at malloc.c:5034 #2 0x00007ffff62a17ec in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738 #3 0x00007ffff7b90fbe in _cfg_lexer_free (ptr=0x625760, yyscanner=0x618720) at cfg-lex.c:2553 #4 0x00007ffff7b90f1e in _cfg_lexer_lex_destroy (yyscanner=0x618720) at cfg-lex.c:2497 #5 0x00007ffff7b57897 in cfg_lexer_free (self=0x61d6e0) at ../../lib/cfg-lexer.c:841 #6 0x00007ffff7b55180 in cfg_run_parser (self=0x617910, lexer=0x61d6e0, parser=0x7ffff7dd9aa0, result=0x7fffffffcd78, arg=0x0) at ../../lib/cfg.c:318 #7 0x00007ffff7b5528d in cfg_read_config (self=0x617910, fname=0x609290 "etc/test.conf", syntax_only=0, preprocess_into=0x0) at ../../lib/cfg.c:347 #8 0x00007ffff7b77ed9 in main_loop_init () at ../../lib/mainloop.c:674 #9 0x0000000000401994 in main (argc=1, argv=0x7fffffffcee8) at ../../syslog-ng/main.c:239 And that's how far I had time to go with this today, I'll try to look into it later, time permitting. -- |8]
On Wed, 2012-03-14 at 08:10 +0100, Gergely Nagy wrote:
Peter Czanik <czanik@balabit.hu> writes:
Hello,
I tried to compile and run alpha1 on openSUSE 12.1 and FreeBSD 8.1. I got a segfault on start on both. Here are the backtraces:
A little more info about this:
$ sbin/syslog-ng syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' WARNING: Configuration file format is too old, please update it to use the 3.4 format as some constructs might operate inefficiently; *** glibc detected *** sbin/syslog-ng: double free or corruption (fasttop): 0x00000000011508e0 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x75ab6)[0x7f2c4318dab6] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f2c431927ec] /home/algernon/install/sng/3.4/lib/syslog-ng/libafmongodb.so(afmongodb_parse+0xf4e)[0x7f2c4021765d] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(+0x4caa8)[0x7f2c44a6eaa8] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(plugin_parse_config+0x174)[0x7f2c44a6f201] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(main_parse+0x108a)[0x7f2c44a83aba] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(+0x23528)[0x7f2c44a45528] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(cfg_run_parser+0x124)[0x7f2c44a46171] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(cfg_read_config+0x84)[0x7f2c44a4628d] /home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(main_loop_init+0xa9)[0x7f2c44a68ed9] sbin/syslog-ng(main+0x1cf)[0x401994] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f2c43136ead] sbin/syslog-ng[0x401549]
Working from here:
#5 0x00007ffff332665d in afmongodb_parse (lexer=0x61d6f0, instance=0x7fffffffa870, arg=0x0) at afmongodb-grammar.y:799 799 | KW_REPLACE '(' string string ')' { value_pairs_transform_set_add_func(last_vp_transset, value_pairs_new_transform_replace($3, $4)); free($3); free($4); }
Something clobbers the heap before the free calls, running under valgrind would probably reveal the cause. However I'd need your configuration to get more information.
I tried playing with GDB and breaking after value_pairs_transform_set_add_func(), to see where things get freed twice, but after a few minutes of not being able to make heads and tails out of the bison-generated code I gave up.
But perhaps the above will shed some light on where the problem may be. The next step I'd take is try to find a version of 3.4 that works, and git bisect.
It does look like the problem is somewhere in the grammar stuff, as both backtraces come from ${module}_parse, and it affects multiple modules.
I don't think so, the one found by Peter is probably related to the new file-perm specifying code, which may not have been properly used in the afsocket module. Now as I was second guessing it, I've found the culprit, it is fixed here: commit daded120cefa7217dac0b1a5d9a870a7fa7fc51d Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Mar 14 09:54:35 2012 +0100 unix-dgram: fixed crash in file permission parsing code A line to set last_file_perm_options was missed in the unix-dgram variant of afsocket, which caused the last_file_perm_options variable to be initialized to NULL, causing a segfault later on. Reported-By: Peter Czanik <peter.czanik@balabit.hu> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
#5 0x00007ffff332665d in afmongodb_parse (lexer=0x61d6f0, instance=0x7fffffffa870, arg=0x0) at afmongodb-grammar.y:799 799 | KW_REPLACE '(' string string ')' { value_pairs_transform_set_add_func(last_vp_transset, value_pairs_new_transform_replace($3, $4)); free($3); free($4); }
Something clobbers the heap before the free calls, running under valgrind would probably reveal the cause.
Valgrind shows this, using the attached config: ==16117== Invalid free() / delete / delete[] / realloc() ==16117== at 0x40279D4: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==16117== by 0x4A7A42D: vp_cmdline_parse_rekey_finish (value-pairs.c:462) ==16117== by 0x4A7AC86: value_pairs_new_from_cmdline (value-pairs.c:681) ==16117== by 0x6FF1CAD: tf_json_prepare (tfjson.c:53) ==16117== by 0x4A75E7E: log_template_add_func_elem (templates.c:801) ==16117== by 0x4A76629: log_template_compile (templates.c:1020) ==16117== by 0x4A4B425: cfg_tree_check_inline_template (cfg-tree.c:984) ==16117== by 0x6DE853F: affile_parse (affile-grammar.y:822) ==16117== by 0x4A6EABB: cfg_parser_parse (cfg-parser.h:83) ==16117== by 0x4A6F214: plugin_parse_config (plugin.c:211) ==16117== by 0x4A83ACD: main_parse (cfg-grammar.y:610) ==16117== by 0x4A45527: cfg_parser_parse (cfg-parser.h:83) ==16117== Address 0x6a38f90 is 0 bytes inside a block of size 7 free'd ==16117== at 0x40279D4: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==16117== by 0x4A7AC57: value_pairs_new_from_cmdline (value-pairs.c:676) ==16117== by 0x6FF1CAD: tf_json_prepare (tfjson.c:53) ==16117== by 0x4A75E7E: log_template_add_func_elem (templates.c:801) ==16117== by 0x4A76629: log_template_compile (templates.c:1020) ==16117== by 0x4A4B425: cfg_tree_check_inline_template (cfg-tree.c:984) ==16117== by 0x6DE853F: affile_parse (affile-grammar.y:822) ==16117== by 0x4A6EABB: cfg_parser_parse (cfg-parser.h:83) ==16117== by 0x4A6F214: plugin_parse_config (plugin.c:211) ==16117== by 0x4A83ACD: main_parse (cfg-grammar.y:610) ==16117== by 0x4A45527: cfg_parser_parse (cfg-parser.h:83) ==16117== by 0x4A46170: cfg_run_parser (cfg.c:316) This is a bug introduced by an earlier patch of mine that removes the --rekey option. I'll sand a patch shortly to fix that. Valgrind would've been my next try, but I had to catch a bus. O:)
However I'd need your configuration to get more information.
My config's now attached, though it's of little use now, as valgrind found the bug above. @version: 3.4 @include "scl.conf" options { threaded(yes); }; source s_local { # system(); internal(); }; destination d_local { file("/tmp/test.json" template("$(format-json --key json.* --rekey --shift 4 --add-prefix JSON --replace JSON.foo=foo --key cee.*)\n")); }; log { source(s_local); destination(d_local); }; -- |8]
On Wed, 2012-03-14 at 10:23 +0100, Gergely Nagy wrote:
Balazs Scheidler <bazsi@balabit.hu> writes:
#5 0x00007ffff332665d in afmongodb_parse (lexer=0x61d6f0, instance=0x7fffffffa870, arg=0x0) at afmongodb-grammar.y:799 799 | KW_REPLACE '(' string string ')' { value_pairs_transform_set_add_func(last_vp_transset, value_pairs_new_transform_replace($3, $4)); free($3); free($4); }
Something clobbers the heap before the free calls, running under valgrind would probably reveal the cause.
Valgrind shows this, using the attached config:
==16117== Invalid free() / delete / delete[] / realloc() ==16117== at 0x40279D4: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==16117== by 0x4A7A42D: vp_cmdline_parse_rekey_finish (value-pairs.c:462) ==16117== by 0x4A7AC86: value_pairs_new_from_cmdline (value-pairs.c:681) ==16117== by 0x6FF1CAD: tf_json_prepare (tfjson.c:53) ==16117== by 0x4A75E7E: log_template_add_func_elem (templates.c:801) ==16117== by 0x4A76629: log_template_compile (templates.c:1020) ==16117== by 0x4A4B425: cfg_tree_check_inline_template (cfg-tree.c:984) ==16117== by 0x6DE853F: affile_parse (affile-grammar.y:822) ==16117== by 0x4A6EABB: cfg_parser_parse (cfg-parser.h:83) ==16117== by 0x4A6F214: plugin_parse_config (plugin.c:211) ==16117== by 0x4A83ACD: main_parse (cfg-grammar.y:610) ==16117== by 0x4A45527: cfg_parser_parse (cfg-parser.h:83) ==16117== Address 0x6a38f90 is 0 bytes inside a block of size 7 free'd ==16117== at 0x40279D4: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==16117== by 0x4A7AC57: value_pairs_new_from_cmdline (value-pairs.c:676) ==16117== by 0x6FF1CAD: tf_json_prepare (tfjson.c:53) ==16117== by 0x4A75E7E: log_template_add_func_elem (templates.c:801) ==16117== by 0x4A76629: log_template_compile (templates.c:1020) ==16117== by 0x4A4B425: cfg_tree_check_inline_template (cfg-tree.c:984) ==16117== by 0x6DE853F: affile_parse (affile-grammar.y:822) ==16117== by 0x4A6EABB: cfg_parser_parse (cfg-parser.h:83) ==16117== by 0x4A6F214: plugin_parse_config (plugin.c:211) ==16117== by 0x4A83ACD: main_parse (cfg-grammar.y:610) ==16117== by 0x4A45527: cfg_parser_parse (cfg-parser.h:83) ==16117== by 0x4A46170: cfg_run_parser (cfg.c:316)
This is a bug introduced by an earlier patch of mine that removes the --rekey option. I'll sand a patch shortly to fix that.
Valgrind would've been my next try, but I had to catch a bus. O:)
However I'd need your configuration to get more information.
My config's now attached, though it's of little use now, as valgrind found the bug above.
I've commited this for the double free: commit cf193a52e2177641921e6ff7cda48bc4a37d971b Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Mar 14 20:47:19 2012 +0100 value-pairs: fixed double free in case of an argument parsing failure Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> and this to report template compilation errors properly: commit 7c2cc16233a5fc21232ec22f56a9ae9022e240bd Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Mar 14 20:48:09 2012 +0100 cfg-tree: handle template compilation errors properly In case a template was specified directly at a destination, its syntax errors were not properly reported while the configuration was being parsed. This patch fixes that. Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
commit 7c2cc16233a5fc21232ec22f56a9ae9022e240bd Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Mar 14 20:48:09 2012 +0100
cfg-tree: handle template compilation errors properly
In case a template was specified directly at a destination, its syntax errors were not properly reported while the configuration was being parsed. This patch fixes that.
Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
Thanks! I wanted to ask for this many times in the past when I had to debug a typo in my templates, but always forgot :| -- |8]
On 03/14/2012 09:55 AM, Balazs Scheidler wrote:
. I don't think so, the one found by Peter is probably related to the new file-perm specifying code, which may not have been properly used in the afsocket module.
Now as I was second guessing it, I've found the culprit, it is fixed here:
commit daded120cefa7217dac0b1a5d9a870a7fa7fc51d Author: Balazs Scheidler<bazsi@balabit.hu> Date: Wed Mar 14 09:54:35 2012 +0100 Just verified, the segfault is gone and alpha1 starts now also on FreeBSD. I'll check the json part of the port next. Bye,
-- Peter Czanik (CzP)<czanik@balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/
On Tue, 2012-03-13 at 21:31 +0100, Peter Czanik wrote:
And another backtrace, this time from openSUSE:
(gdb) backtrace #0 0xb7684e8d in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #1 0xb7684f60 in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #2 0xb7685557 in cfg_tree_compile_rule () from /lib/libsyslog-ng-3.4.0alpha1.so #3 0xb7685954 in cfg_tree_compile () from /lib/libsyslog-ng-3.4.0alpha1.so #4 0xb76859ee in cfg_tree_start () from /lib/libsyslog-ng-3.4.0alpha1.so #5 0xb76808c0 in cfg_init () from /lib/libsyslog-ng-3.4.0alpha1.so #6 0xb76a07ea in main_loop_init () from /lib/libsyslog-ng-3.4.0alpha1.so #7 0x080491a7 in main () (gdb)
Bye,
This one is fixed by this patch, it is triggered by an empty source/destination definition, which seems to be present in the configuration file you use: commit 2d7339985ad8ce9b8d162ca95b3b59d711c27ad2 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Mar 14 09:36:20 2012 +0100 cfg-tree: fixed segfault triggered by empty source/destination object When a source/destination object is empty a segfault happened during the compilation of configuration (for sources) or at the first message (for destinations). This was fixed by handling such cases. Reported-By: Peter Czanik <czp@balabit.hu> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Bazsi
Hello, On 03/14/2012 09:37 AM, Balazs Scheidler wrote:
This one is fixed by this patch, it is triggered by an empty source/destination definition, which seems to be present in the configuration file you use: Thanks, it seems to be fixed now. Bye,
-- Peter Czanik (CzP)<czanik@balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/
On Tue, 2012-03-13 at 21:31 +0100, Peter Czanik wrote:
Hello,
I tried to compile and run alpha1 on openSUSE 12.1 and FreeBSD 8.1. I got a segfault on start on both. Here are the backtraces:
fb81# gdb74 -c syslog-ng.core /usr/local/sbin/syslog-ng GNU gdb (GDB) 7.4 [GDB v7.4 for FreeBSD] Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd8.1". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done. [New process 100055] [New Thread 8018041c0 (LWP 100055)] Core was generated by `syslog-ng'. Program terminated with signal 11, Segmentation fault. #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 60 file-perms.c: No such file or directory. (gdb) backtrace #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 #1 0x0000000801c0cda0 in afsocket_parse (lexer=0x80196b000, instance=0x7fffffffc788, arg=0x0) at afsocket-grammar.y:1186 #2 0x000000080068b4f3 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=0x80196b000, self=0x801d16fc0) at cfg-parser.h:83 #3 plugin_parse_config (self=0x801d174b0, cfg=<optimized out>, yylloc=<optimized out>, arg=0x0) at plugin.c:211 #4 0x0000000800698730 in main_parse (lexer=0x80196b000, dummy=0x7fffffffec48, arg=0x0) at cfg-grammar.y:518 #5 0x0000000800669ef9 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=<optimized out>, self=<optimized out>) at cfg-parser.h:83 #6 cfg_run_parser (self=0x801949180, lexer=0x80196b000, parser=0x8007c4140, result=0x7fffffffec48, arg=0x801c0f760) at cfg.c:316 #7 0x000000080066a006 in cfg_read_config (self=0x801949180, fname=0x8006ab16b "/usr/local/etc/syslog-ng.conf", syntax_only=<optimized out>, preprocess_into=0x0) at cfg.c:347 #8 0x0000000800685baa in main_loop_init () at mainloop.c:674 #9 0x000000000040171f in main () (gdb)
This is either a linking related problem, or I'm blind while staring at the source code. Can you post your configuration file too?
This binary was compiled by copying sysutils/syslog-ng port to syslog-ng-devel, adding --enable-debug to configure args. I ran into one more interesting thing: with json-c enabled in "make config", configure failed with:
configure: error: Cannot find json-c version >= 0.9: is pkg-config in path?
When disabled json-c in "make config" it was found by configure and was built. The relevant part of "Makefile" is:
.if defined(WITH_JSON_GLIB) LIB_DEPENDS+= json-glib-1.0.0:${PORTSDIR}/devel/json-glib CONFIGURE_ARGS+= --enable-json --with-json=json-glib PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .elif defined(WITH_JSON_C) LIB_DEPENDS+= json.0:${PORTSDIR}/devel/json-c CONFIGURE_ARGS+= --enable-json --with-json=json-c PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .else CONFIGURE_ARGS+= --disable-json .endif
So, in theory json support was disabled...
hmm.. there's no such switch as --enable-json, although I'd tend to agree that this is confusing. The reason is that we support two json implementations and that's usually specified using --with-json=<xxx>, and once that is supplied --enable-json is not needed. So to disable json support you'd need to use --without-json, but suggestions how this should work is appreciated. -- Bazsi
Hello, On 03/14/2012 09:50 AM, Balazs Scheidler wrote:
I ran into one more interesting thing: with json-c enabled in "make config", configure failed with:
configure: error: Cannot find json-c version>= 0.9: is pkg-config in path?
When disabled json-c in "make config" it was found by configure and was built. The relevant part of "Makefile" is:
.if defined(WITH_JSON_GLIB) LIB_DEPENDS+= json-glib-1.0.0:${PORTSDIR}/devel/json-glib CONFIGURE_ARGS+= --enable-json --with-json=json-glib PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .elif defined(WITH_JSON_C) LIB_DEPENDS+= json.0:${PORTSDIR}/devel/json-c CONFIGURE_ARGS+= --enable-json --with-json=json-c PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .else CONFIGURE_ARGS+= --disable-json .endif
So, in theory json support was disabled... hmm.. there's no such switch as --enable-json, although I'd tend to agree that this is confusing. The reason is that we support two json implementations and that's usually specified using --with-json=<xxx>, and once that is supplied --enable-json is not needed. OK, I fixed the ports Makefile. Still, when I enable json-c in make conf, it fails with the above error.
So to disable json support you'd need to use --without-json, but suggestions how this should work is appreciated. "--with-json=none" would be a nice pair to "auto". It would be nice, if it could be explicitly disabled, as if json is available on the system, it's automagically used. So if it's there as a dependency of another software and later deleted, it would render syslog-ng unusable due to missing dependencies. Bye,
-- Peter Czanik (CzP)<czanik@balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/
Peter Czanik <czanik@balabit.hu> writes:
So, in theory json support was disabled... hmm.. there's no such switch as --enable-json, although I'd tend to agree that this is confusing. The reason is that we support two json implementations and that's usually specified using --with-json=<xxx>, and once that is supplied --enable-json is not needed. OK, I fixed the ports Makefile. Still, when I enable json-c in make conf, it fails with the above error.
I'll look into this at a later time, my FreeBSD refuses to boot at the moment.
So to disable json support you'd need to use --without-json, but suggestions how this should work is appreciated. "--with-json=none" would be a nice pair to "auto". It would be nice, if it could be explicitly disabled, as if json is available on the system, it's automagically used. So if it's there as a dependency of another software and later deleted, it would render syslog-ng unusable due to missing dependencies.
--with-json=no should work, as should --without-json, but I'll run a test in a bit. -- |8]
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
Peter Czanik