syslog-ng 3.3.0beta & ESX crashes
HI there, I've have a syslog-ng 3.3.0beta1 (at least the tag I get from the git build) that keeps falling over as it receives logs from ESX servers, this is on a Red Hat Enterprise Linux Server release 6.0 (Santiago) inside a VM. The dmesg output gives "interesting" responses: syslog-ng[1371]: segfault at 4e1fe858 ip 000000004e1fe858 sp 00007fff8d7b7958 error 14 in libselinux.so.1[3841600000+1d000] syslog-ng[2054]: segfault at 4e1fe992 ip 000000004e1fe992 sp 00007fff8d7b7958 error 14 in libselinux.so.1[3841600000+1d000] syslog-ng[2080] general protection ip:7f42ab38c911 sp:7fff8d7b7a20 error:0 in libaffile.so[7f42ab388000+10000] syslog-ng[2196]: segfault at 4e1feb26 ip 000000004e1feb26 sp 00007fff8d7b7a18 error 14 in libselinux.so.1[3841600000+1d000] syslog-ng[2281]: segfault at 1d2a930 ip 0000000001d2a930 sp 00007fff8d7b7958 error 15 syslog-ng[2303]: segfault at 4e200944 ip 000000004e200944 sp 00007fff8d7b7a18 error 14 in libselinux.so.1[3841600000+1d000] syslog-ng[2346] general protection ip:7f42ab38c911 sp:7fff8d7b7960 error:0 in libaffile.so[7f42ab388000+10000] syslog-ng[2354] general protection ip:7f42ab38c911 sp:7fff8d7b7a20 error:0 in libaffile.so[7f42ab388000+10000] syslog-ng[2366] general protection ip:7f42ab38c911 sp:7fff8d7b7960 error:0 in libaffile.so[7f42ab388000+10000] syslog-ng[2388]: segfault at 1e350c8 ip 0000000001e350c8 sp 00007fff8d7b7958 error 15 syslog-ng[2397] general protection ip:7f42ab38c911 sp:7fff8d7b7a20 error:0 in libaffile.so[7f42ab388000+10000] syslog-ng[2418]: segfault at 34312e35ba ip 00007f42ab7e921f sp 00007fff8d7b7900 error 4 in libsyslog-ng.so.0.0.0[7f42ab7ac000+91000] syslog-ng[2426] general protection ip:7f42ab38c911 sp:7fff8d7b7960 error:0 in libaffile.so[7f42ab388000+10000] syslog-ng[2458]: segfault at 1d2cac8 ip 0000000001d2cac8 sp 00007fff8d7b7a38 error 15 syslog-ng[2487]: segfault at bb ip 00000000000000bb sp 00007fff8d7b7a18 error 14 in syslog-ng[400000+2000] [visagehe@tsysl01 ~]$ Not always, but several times I also see these in the internal syslog-ng log output: Jul 15 11:17:56 tsysl01 syslog-ng[2066]: syslog-ng starting up; version='3.3.0beta1' Jul 15 11:19:49 10.122.122.14 syslog-ng[2066]: Error processing log message: </memoryAllocation></obj> Jul 15 11:20:08 tsysl01 syslog-ng[2080]: syslog-ng starting up; version='3.3.0beta1' Jul 15 11:21:49 10.122.122.14 syslog-ng[2080]: Error processing log message: </memoryAllocation></obj> Jul 15 11:22:25 tsysl01 syslog-ng[2196]: syslog-ng starting up; version='3.3.0beta1' Jul 15 11:24:58 10.122.122.14 syslog-ng[2196]: Error processing log message: </memoryAllocation></obj> Jul 15 11:25:28 tsysl01 syslog-ng[2274]: syslog-ng starting up; version='3.3.0beta1' It appears that the ESX servers are sending big streams of console usage information to the syslog-ng (on UDP) which appears to obviously not always fit in standard syslog-ng packet formats :( Any advice on fixing this, or perhaps going back to a previous version of syslog-ng?? Thank you Hendrik
Hendrik Visage <hvjunk@gmail.com> writes:
I've have a syslog-ng 3.3.0beta1 (at least the tag I get from the git build) that keeps falling over as it receives logs from ESX servers, this is on a Red Hat Enterprise Linux Server release 6.0 (Santiago) inside a VM. The dmesg output gives "interesting" responses:
[...snip...] Would it be possible to compile syslog-ng with debugging symbols (CFLAGS="-g" ./configure --enable-debug), and run it with --enable-core? Looking at the backtrace, we could perhaps know a bit more about why syslog-ng is segfaulting. Other than that, I'd suggest either going back to 3.2, or trying the latest version of 3.3 from git head, as there have been quite a lot of stability fixes since beta1. -- |8]
On Fri, Jul 15, 2011 at 12:21 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hendrik Visage <hvjunk@gmail.com> writes:
I've have a syslog-ng 3.3.0beta1 (at least the tag I get from the git build) that keeps falling over as it receives logs from ESX servers, this is on a Red Hat Enterprise Linux Server release 6.0 (Santiago) inside a VM. The dmesg output gives "interesting" responses:
[...snip...]
Would it be possible to compile syslog-ng with debugging symbols (CFLAGS="-g" ./configure --enable-debug), and run it with --enable-core?
Done.
Other than that, I'd suggest either going back to 3.2, or trying the latest version of 3.3 from git head, as there have been quite a lot of stability fixes since beta1.
Jul 15 15:47:09 tsysl01 syslog-ng[27978]: syslog-ng starting up; version='3.3.0beta1' Just "which beta1 is this that I've downloaded from githead???" comes to mind :( Perhaps embed a githead version for the git repositories ;) Hendrik
Hendrik Visage <hvjunk@gmail.com> writes:
On Fri, Jul 15, 2011 at 12:21 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hendrik Visage <hvjunk@gmail.com> writes:
I've have a syslog-ng 3.3.0beta1 (at least the tag I get from the git build) that keeps falling over as it receives logs from ESX servers, this is on a Red Hat Enterprise Linux Server release 6.0 (Santiago) inside a VM. The dmesg output gives "interesting" responses:
[...snip...]
Would it be possible to compile syslog-ng with debugging symbols (CFLAGS="-g" ./configure --enable-debug), and run it with --enable-core?
Done.
A backtrace of the crash would then be useful to have. O:) ,---- | $ gdb -c /path/to/core-file /path/to/syslog-ng | (gdb) bt full `----
Other than that, I'd suggest either going back to 3.2, or trying the latest version of 3.3 from git head, as there have been quite a lot of stability fixes since beta1.
Jul 15 15:47:09 tsysl01 syslog-ng[27978]: syslog-ng starting up; version='3.3.0beta1'
Just "which beta1 is this that I've downloaded from githead???" comes to mind :(
Perhaps embed a githead version for the git repositories ;)
That's a bit tricky, but you're right - it would be nice to have an indication about what revision the binary was compiled from. -- |8]
Hi Gergely, Looks like I have a syslog-ng 3.3beta1 (fetched from GITHEAD) that is having a serious memory leak (and I'm not yet pushing all our logs to it ;( Any advice in how to help you track this problem? Hendrik On Fri, Jul 15, 2011 at 4:06 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hendrik Visage <hvjunk@gmail.com> writes:
On Fri, Jul 15, 2011 at 12:21 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hendrik Visage <hvjunk@gmail.com> writes:
I've have a syslog-ng 3.3.0beta1 (at least the tag I get from the git build) that keeps falling over as it receives logs from ESX servers, this is on a Red Hat Enterprise Linux Server release 6.0 (Santiago) inside a VM. The dmesg output gives "interesting" responses:
[...snip...]
Would it be possible to compile syslog-ng with debugging symbols (CFLAGS="-g" ./configure --enable-debug), and run it with --enable-core?
Done.
A backtrace of the crash would then be useful to have. O:)
,---- | $ gdb -c /path/to/core-file /path/to/syslog-ng | (gdb) bt full `----
Other than that, I'd suggest either going back to 3.2, or trying the latest version of 3.3 from git head, as there have been quite a lot of stability fixes since beta1.
Jul 15 15:47:09 tsysl01 syslog-ng[27978]: syslog-ng starting up; version='3.3.0beta1'
Just "which beta1 is this that I've downloaded from githead???" comes to mind :(
Perhaps embed a githead version for the git repositories ;)
That's a bit tricky, but you're right - it would be nice to have an indication about what revision the binary was compiled from.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hendrik Visage <hvjunk@gmail.com> writes:
Hi Gergely,
Looks like I have a syslog-ng 3.3beta1 (fetched from GITHEAD) that is having a serious memory leak (and I'm not yet pushing all our logs to it ;(
Any advice in how to help you track this problem?
Hrm. Well, valgrind would be best, but that slows things down to almost a grounding halt. (But in case you want to try that: G_SLICE=always-malloc valgrind --leak-check=full -v --show-reachable-yes --log-file=valgrind.log syslog-ng -F would be the command to run ;) Other than that, a config could be useful, so I can try and reproduce the problem (or at least identify possible areas where the leak might be coming from, and go from there), and see if I can fix it. -- |8]
participants (2)
-
Gergely Nagy
-
Hendrik Visage