Yes, you are absolutely right. I can see the infinite loop (with brk() call inside) from the tracing file. Thank you so much for your great help which saved me a lot of time. I was thinking something wrong when I built glib :)
To: syslog-ng@lists.balabit.hu Date: Tue, 15 Dec 2009 18:04:24 +0100 Subject: Re: [syslog-ng] syslog-ng OSE 3.0.4 coredumps when use "internal()" in s_local
On Fri, 2009-12-11 at 18:21 +0000, nova club wrote:
Thank you so much for your attention. I tried 3.0.5 and it has same problem. Please see the attached tarball consisting of a conf file and static-linked syslog-ng binary.
The platform I am using is "6.4-RELEASE-p6 FreeBSD 6.4-RELEASE-p6 i386". The command line is like "syslog-ng -f <conf> -d -F". And for now, I have not found any special or abnormal resource "limit" setup on that box.
Another interesting thing is no coredumps if I use "-e" option too.
The conf file actually is a copy of "contrib/freebsd-packaging/syslog-ng.conf.example", I just added a new log path for a program called "hecxd".
Thanks a lot again.
I don't have any more time, but the problem seems to be caused by the fact that --debug messages are enabled.
filtering generates a 'debug' message, which in turn is processed by the same filter generating another filter and so on.
So effectively syslog-ng is busy processing its own message. And these messages are collected in memory (possibly in a destination queue), thus syslog-ng is busy allocating memory too.
At the end of the day, the memory runs out, causing an abort, which will cause a segfault.
There's explicit processing in syslog-ng to stop such message loops caused by internal(), but this is clearly not triggering this case.
Disabling --debug messages should resolve your problem though.
From: bazsi@balabit.hu To: syslog-ng@lists.balabit.hu Date: Fri, 11 Dec 2009 07:52:02 +0100 Subject: Re: [syslog-ng] syslog-ng OSE 3.0.4 coredumps when use "internal()" in s_local
On Wed, 2009-12-09 at 18:04 +0000, nova club wrote:
Hi there,
I am a new user of syslog-ng. Thanks for your time and attention in advance.
The problem is syslog-ng coredumps when use "internal()" in "source", however it will work fine without "internal()". The conf file is like:
hmm... g_malloc() never fails, it aborts() instead after logging the failure using assert.
This assert is then caught by syslog-ng, to be put in the internal() message source , but then again allocation fails, thus the segfault.
Are you sure that syslog-ng is not severely constrained by ulimit or something?
Generally if memory allocation fails, syslog-ng aborts.
Ah, as I see you said the core file is large, this seems to be some kind of memory leak, or overuse of memory, and that's the root reason of the crash.
Can you send me your complete configuration file? Maybe we could reproduce the problem. Do you happen to have large log_fifo_size() configured to one of your destinations?
===>BEGIN @version:3.0
options { dir_perm(0755); perm(0644); chain_hostnames(no); keep_hostname(yes); };
# Default local source. source local { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); };
...
filter f_hecxd { program("hecxd"); };
...
log { source(local); filter(f_hecxd); destination(d_hecxd); }; ==>END
The options used for building syslog-ng is:
CFLAGS='-Wl,-Bdynamic -liconv -Wl,-Bstatic'
--enable-static-linking
\ --enable-debug \ --enable-ssl --enable-ipv6=no --enable-tcp-wrapper=no \ --with-pidfile-dir="/var/run/"
The platform is: FREEBSD 6.4-RELEASE-p6 i386
I got a very big core dump file (>500M), by backtracing that file, I can see the problem is "g_malloc0" for some reason failed to allocate memory, while syslog-ng did not check the return value and tried to use the "0x0" pointer in a "strlen()" call later. Here is a part of trace information:
#0 0x080a2d8d in strlen () No symbol table info available. #1 0x0804e7fe in log_msg_new_internal (prio=43, msg=0x0, flags=6) at logmsg.c:1971 buf = (gchar *) 0x0 self = (LogMessage *) 0x8159000 #2 0x080492b1 in msg_send_internal_message (prio=43, msg=0x0) at messages.c:128 context = (MsgContext *) 0x81893a0 m = (LogMessage *) 0x7 #3 0x0804954c in msg_log_func (log_domain=0x81186ea "GLib", log_flags=6, msg=0x0, user_data=0x0) at messages.c:187 pri = 43 #4 0x08084906 in IA__g_logv (log_domain=0x81186ea "GLib", log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to allocate %u bytes", args1=0xbfbfe6cc "3�\021\bl") at gmessages.c:519 msg = (gchar *) 0x0 was_fatal = 0 was_recursion = 0 i = 2 #5 0x08084ae4 in IA__g_log (log_domain=0x81186ea "GLib", log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to allocate %u bytes") at gmessages.c:569 No locals. #6 0x08084239 in IA__g_malloc0 (n_bytes=108) at gmem.c:155 mem = 0x0 #7 0x0804e5ef in log_msg_new ( msg=0x8a82180 "<47> syslog-ng[3277]: Filter node evaluation result; filter_result='not-match', filter_type='level'\n", length=100, saddr=0x0, flags=6, bad_hostname=0x0, assume_timezone=-1, default_pri=47) at logmsg.c:1873 self = (LogMessage *) 0x810ba1b #8 0x0804e80a in log_msg_new_internal (prio=47, msg=0x817f500 "Filter node evaluation result; filter_result='not-match', filter_type='level'", flags=6) at logmsg.c:1971 buf = (gchar *) 0x8a82180 "<47> syslog-ng[3277]: Filter node evaluation result; filter_result='not-match', filter_type='level'\n" self = (LogMessage *) 0x8159000 #9 0x080492b1 in msg_send_internal_message (prio=47, msg=0x817f500 "Filter node evaluation result; filter_result='not-match', filter_type='level'") at messages.c:128 context = (MsgContext *) 0x81893a0 m = (LogMessage *) 0x8182030
If you need the completed coredump file, just let me know and tell me where to put it :)
Any help is much appreciated.
Thanks a lot. Shen
Windows Live: Make it easier for your friends to see what you’re up to on Facebook.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________ Get a great deal on Windows 7 and see how it works the way you want. Check out the offers on Windows 7now. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
_________________________________________________________________ Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 now http://go.microsoft.com/?linkid=9691818