[syslog-ng] segfault in 3.4 alpha1

[Gergely Nagy]

Peter Czanik <czanik at balabit.hu> writes:

> Hello,
>
> I tried to compile and run alpha1 on openSUSE 12.1 and FreeBSD 8.1. I 
> got a segfault on start on both. Here are the backtraces:

A little more info about this:

$ sbin/syslog-ng
syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted'
WARNING: Configuration file format is too old, please update it to use the 3.4 format as some constructs might operate inefficiently;
*** glibc detected *** sbin/syslog-ng: double free or corruption (fasttop): 0x00000000011508e0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x75ab6)[0x7f2c4318dab6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f2c431927ec]
/home/algernon/install/sng/3.4/lib/syslog-ng/libafmongodb.so(afmongodb_parse+0xf4e)[0x7f2c4021765d]
/home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(+0x4caa8)[0x7f2c44a6eaa8]
/home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(plugin_parse_config+0x174)[0x7f2c44a6f201]
/home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(main_parse+0x108a)[0x7f2c44a83aba]
/home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(+0x23528)[0x7f2c44a45528]
/home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(cfg_run_parser+0x124)[0x7f2c44a46171]
/home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(cfg_read_config+0x84)[0x7f2c44a4628d]
/home/algernon/install/sng/3.4/lib/libsyslog-ng-3.4.0alpha0.so(main_loop_init+0xa9)[0x7f2c44a68ed9]
sbin/syslog-ng(main+0x1cf)[0x401994]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f2c43136ead]
sbin/syslog-ng[0x401549]

Working from here:

#5  0x00007ffff332665d in afmongodb_parse (lexer=0x61d6f0, instance=0x7fffffffa870, arg=0x0) at afmongodb-grammar.y:799
799		| KW_REPLACE '(' string string ')' { value_pairs_transform_set_add_func(last_vp_transset, value_pairs_new_transform_replace($3, $4)); free($3); free($4); }

I tried playing with GDB and breaking after
value_pairs_transform_set_add_func(), to see where things get freed
twice, but after a few minutes of not being able to make heads and tails
out of the bison-generated code I gave up.

But perhaps the above will shed some light on where the problem may
be. The next step I'd take is try to find a version of 3.4 that works,
and git bisect.

It does look like the problem is somewhere in the grammar stuff, as both
backtraces come from ${module}_parse, and it affects multiple modules.

The interesting thing is that make check still works, which suggests
there is something in our config that triggers the bug.

Furthermore, if I remove the @module lines from my config, I get a
proper segfault:

Program received signal SIGSEGV, Segmentation fault.
malloc_consolidate (av=0x7ffff65a9e60) at malloc.c:5155
5155	malloc.c: No such file or directory.
(gdb) bt
#0  malloc_consolidate (av=0x7ffff65a9e60) at malloc.c:5155
#1  0x00007ffff629f214 in _int_malloc (av=0x7ffff65a9e60, bytes=1221) at malloc.c:4373
#2  0x00007ffff62a0eb2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:4065
#3  0x00007ffff7de780f in ?? () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff7de2fb6 in ?? () from /lib64/ld-linux-x86-64.so.2
#5  0x00007ffff7de46b8 in ?? () from /lib64/ld-linux-x86-64.so.2
#6  0x00007ffff7deee5e in ?? () from /lib64/ld-linux-x86-64.so.2
#7  0x00007ffff7dea926 in ?? () from /lib64/ld-linux-x86-64.so.2
#8  0x00007ffff7dee89a in ?? () from /lib64/ld-linux-x86-64.so.2
#9  0x00007ffff67caf66 in dlopen_doit (a=<optimized out>) at dlopen.c:67
#10 0x00007ffff7dea926 in ?? () from /lib64/ld-linux-x86-64.so.2
#11 0x00007ffff67cb2ec in _dlerror_run (operate=0x7ffff67caf00 <dlopen_doit>, args=0x7fffffffa6a0) at dlerror.c:164
#12 0x00007ffff67caee1 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:88
#13 0x00007ffff7717944 in _g_module_open (bind_local=<optimized out>, bind_lazy=<optimized out>, file_name=0x6e1900 "/home/algernon/install/sng/3.4/lib/syslog-ng/libafmongodb.so")
    at /tmp/buildd/glib2.0-2.30.2/./gmodule/gmodule-dl.c:99
#14 g_module_open (file_name=0x618e10 "/home/algernon/install/sng/3.4/lib/syslog-ng/libafmongodb.so", flags=<optimized out>) at /tmp/buildd/glib2.0-2.30.2/./gmodule/gmodule.c:439
#15 0x00007ffff7b7e4d4 in plugin_dlopen_module (module_name=0x648280 "afmongodb", module_path=0x618a30 "/home/algernon/install/sng/3.4/lib/syslog-ng") at ../../lib/plugin.c:305
#16 0x00007ffff7b7e626 in plugin_load_module (module_name=0x648280 "afmongodb", cfg=0x617900, args=0x0) at ../../lib/plugin.c:349
#17 0x00007ffff7b7df56 in plugin_find (cfg=0x617900, plugin_type=2, plugin_name=0x6e0eb0 "mongodb") at ../../lib/plugin.c:148
#18 0x00007ffff7b92a1d in main_parse (lexer=0x61d6f0, dummy=0x7fffffffcd98, arg=0x0) at cfg-grammar.y:607
#19 0x00007ffff7b54528 in cfg_parser_parse (self=0x7ffff7dd9aa0, lexer=0x61d6f0, instance=0x7fffffffcd98, arg=0x0) at ../../lib/cfg-parser.h:83
#20 0x00007ffff7b55171 in cfg_run_parser (self=0x617900, lexer=0x61d6f0, parser=0x7ffff7dd9aa0, result=0x7fffffffcd98, arg=0x0) at ../../lib/cfg.c:316
#21 0x00007ffff7b5528d in cfg_read_config (self=0x617900, fname=0x7ffff7baa960 "/home/algernon/install/sng/3.4/etc/syslog-ng.conf", syntax_only=0, preprocess_into=0x0) at ../../lib/cfg.c:347
#22 0x00007ffff7b77ed9 in main_loop_init () at ../../lib/mainloop.c:674
#23 0x0000000000401994 in main (argc=1, argv=0x7fffffffcf08) at ../../syslog-ng/main.c:239

Earlier:

Module loaded and initialized successfully; module='confgen'
Finishing include; filename='/home/algernon/install/sng/3.4/share/include/scl/syslogconf/plugin.conf', depth='2'
Finishing include; filename='/home/algernon/install/sng/3.4/etc/scl.conf', depth='1'
Module loaded and initialized successfully; module='afsocket-tls'
Module loaded and initialized successfully; module='jsonparser'
Module loaded and initialized successfully; module='affile'
Module loaded and initialized successfully; module='tfjson'

Using a simpler config like this:

@version: 3.4
@include "scl.conf"

options {
	threaded(yes);
};

source s_local {
#	system();
	internal();
};

destination d_local {
	file("/tmp/test.json" template("$(format-json --key json.* --rekey --shift 4 --add-prefix JSON --replace JSON.foo=foo --key cee.*)\n"));
};

log {
	source(s_local);
	destination(d_local);
};

I get the following backtrace:

(gdb) bt
#0  malloc_consolidate (av=0x7ffff65a9e60) at malloc.c:5155
#1  0x00007ffff629e3e8 in _int_free (av=0x7ffff65a9e60, p=0x621740) at malloc.c:5034
#2  0x00007ffff62a17ec in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738
#3  0x00007ffff7b90fbe in _cfg_lexer_free (ptr=0x625760, yyscanner=0x618720) at cfg-lex.c:2553
#4  0x00007ffff7b90f1e in _cfg_lexer_lex_destroy (yyscanner=0x618720) at cfg-lex.c:2497
#5  0x00007ffff7b57897 in cfg_lexer_free (self=0x61d6e0) at ../../lib/cfg-lexer.c:841
#6  0x00007ffff7b55180 in cfg_run_parser (self=0x617910, lexer=0x61d6e0, parser=0x7ffff7dd9aa0, result=0x7fffffffcd78, arg=0x0) at ../../lib/cfg.c:318
#7  0x00007ffff7b5528d in cfg_read_config (self=0x617910, fname=0x609290 "etc/test.conf", syntax_only=0, preprocess_into=0x0) at ../../lib/cfg.c:347
#8  0x00007ffff7b77ed9 in main_loop_init () at ../../lib/mainloop.c:674
#9  0x0000000000401994 in main (argc=1, argv=0x7fffffffcee8) at ../../syslog-ng/main.c:239

And that's how far I had time to go with this today, I'll try to look
into it later, time permitting.

-- 
|8]