On Sat, 2011-12-31 at 03:39 +0300, Luis Pugoy wrote:

Message: 1 Date: Mon, 5 Dec 2011 19:54:42 +0800 From: Luis Pugoy <lpugoy@insynchq.com> Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu 10.10 To: syslog-ng@lists.balabit.hu Message-ID: <CA +WGxEhKYMQXbYTVUZVScjaXDdjRETGSCk5nZaisW9p-UOQWoA@mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1"

> > Message: 3 > Date: Mon, 05 Dec 2011 10:10:40 +0100 > From: Gergely Nagy <algernon@balabit.hu> > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > 10.10 > To: Syslog-ng users' and developers' mailing list > <syslog-ng@lists.balabit.hu> > Message-ID: <87aa77l6y7.fsf@algernon.balabit> > Content-Type: text/plain > > Luis Pugoy <lpugoy@insynchq.com> writes: > > > Does anyone have any insight into this? The segfault is happening more > > often now. :( > > I'd suggest either upgrading to 3.3 from my repo[1], or if an upgrade is > not feasible, then if you could compile 3.1 with debug symbols, get it > to drop a core, and do a backtrace, that would help tremendously. > > [1]: http://asylum.madhouse-project.org/projects/debian/ > > To do the latter, the following steps should work: > > $ apt-get source syslog-ng > $ sudo apt-get build-dep syslog-ng > $ cd syslog-ng-* > $ DEB_BUILD_OPTIONS="debug nostrip" dpkg-buildpackage -us -uc -rfakeroot > $ sudo dpkg -i ../syslog-ng_*.deb > > Then add --enable-core to SYSLOGNG_OPTS in /etc/default/syslog-ng, and > when it segfaults, it'll make a core in (as far as I remember) /. Once > that happened, do something like this: > > $ gdb -c /core /usr/bin/syslog-ng > (gdb) thread apply all bt full > > This should give us a few hints. > > -- > |8] >

I installed a newer version of the PCRE library. That seems to have worked for now. If it happens again I will be sure to follow your instructions, thanks.

Well, installing another version of the PCRE library did not solve my problem. syslog-ng is still segfaulting. I tried upgrading to 3.2.2 but that did nothing, so I tried to create the backtrace like you said. However I don't think it was helpful in my case. The following is the output of the backtrace:

root@ip-10-34-150-77:/var/lib/syslog-ng# gdb -c core /usr/sbin/syslog-ng GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/syslog-ng...done. [New Thread 14680]

warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /usr/lib/libgthread-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgthread-2.0.so.0 Reading symbols from /lib/libglib-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libglib-2.0.so.0 Reading symbols from /usr/lib/libevtlog.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libevtlog.so.0 Reading symbols from /lib/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libssl.so.0.9.8 Reading symbols from /lib/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.0.9.8 Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/libnet.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnet.so.1 Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libwrap.so.0 Reading symbols from /usr/lib/libdbi.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdbi.so.0 Reading symbols from /lib/libcap.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcap.so.2 Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.3 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libattr.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libattr.so.1 Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_nis.so.2 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_dns.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Core was generated by `/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 (gdb) thread apply all bt full

Thread 1 (Thread 14680): #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 No symbol table info available. Cannot access memory at address 0x7fff4a4e3fd8 (gdb)

If it is in anyway helpful, I think the only usage of PCRE in my syslog-ng.conf is a rewrite rule to change the Unicode character separator line to a new line: subst("\\p{Zl}", "\n", value("MESSAGE") type("pcre") flags("utf8" "global")); Is it related to my problem in some way?

I'm not sure. Can you install the dbgsym debs and regenerate the backtrace that way? There was a bug in syslog-ng when used with a new PCRE, which is not fixed in the 3.1 tree (more exactly a patch is there, but no release was made past 3.1.4). Here's the patch: Author: Balazs Scheidler <bazsi@balabit.hu> 2011-05-03 20:30:48 Committer: Balazs Scheidler <bazsi@balabit.hu> 2011-05-03 20:30:48 Parent: 21a455ecdf808cbd0c57428d1bd3f9feec58419e (csvparser: fixed compilation warning) Child: 9e3e7ee9baccc8565e7866b91a80ee34670ef481 (cfg-grammar.y: fixed the use of uninitialized memory area) Child: 83a461b7ef6351c504786a2a3b51aa9a14e3e9cf (Merge remote branch '3.2/master') Branches: many (43) Follows: v3.2.2 Precedes: v3.3.0beta1 pcre: fixed a potential resource hogging infinite loop when an error occurs Any kind of PCRE error case would cause an infinite loop, when the "global" flag is present and pcre returns an error code. The reported problem is that with PCRE 8.12 we indeed get such an error while doing a global replace. This patch also reworks the way PCRE based replacements are made, that code was hairy, and I just hope this one is one bit less so. One performance related change also made it that improves the speed pcre replacements, which previously zeroed out a 3k array unconditionally in every invocation. Also added some additional testcases to be sure I didn't break anything. Reported-By: Micah Anderson <micah@riseup.net> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Bazsi