<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
Yes, you are absolutely right. I can see the infinite loop (with brk() call inside) from the tracing file. Thank you so much for your great help which saved me a lot of time. I was thinking something wrong when I built glib :)<br><br>> To: syslog-ng@lists.balabit.hu<br>> Date: Tue, 15 Dec 2009 18:04:24 +0100<br>> Subject: Re: [syslog-ng] syslog-ng OSE 3.0.4 coredumps when use "internal()" in s_local<br>> <br>> On Fri, 2009-12-11 at 18:21 +0000, nova club wrote:<br>> > Thank you so much for your attention. I tried 3.0.5 and it has same<br>> > problem. Please see the attached tarball consisting of a conf file and<br>> > static-linked syslog-ng binary.<br>> > <br>> > The platform I am using is "6.4-RELEASE-p6 FreeBSD 6.4-RELEASE-p6<br>> > i386". The command line is like "syslog-ng -f <conf> -d -F". And for<br>> > now, I have not found any special or abnormal resource "limit" setup<br>> > on that box. <br>> > <br>> > Another interesting thing is no coredumps if I use "-e" option too.<br>> > <br>> > The conf file actually is a copy of<br>> > "contrib/freebsd-packaging/syslog-ng.conf.example", I just added a new<br>> > log path for a program called "hecxd".<br>> > <br>> > Thanks a lot again.<br>> <br>> I don't have any more time, but the problem seems to be caused by the<br>> fact that --debug messages are enabled.<br>> <br>> filtering generates a 'debug' message, which in turn is processed by the<br>> same filter generating another filter and so on.<br>> <br>> So effectively syslog-ng is busy processing its own message. And these<br>> messages are collected in memory (possibly in a destination queue), thus<br>> syslog-ng is busy allocating memory too.<br>> <br>> At the end of the day, the memory runs out, causing an abort, which will<br>> cause a segfault.<br>> <br>> There's explicit processing in syslog-ng to stop such message loops<br>> caused by internal(), but this is clearly not triggering this case.<br>> <br>> Disabling --debug messages should resolve your problem though.<br>> <br>> > <br>> > > From: bazsi@balabit.hu<br>> > > To: syslog-ng@lists.balabit.hu<br>> > > Date: Fri, 11 Dec 2009 07:52:02 +0100<br>> > > Subject: Re: [syslog-ng] syslog-ng OSE 3.0.4 coredumps when use<br>> > "internal()" in s_local<br>> > > <br>> > > On Wed, 2009-12-09 at 18:04 +0000, nova club wrote:<br>> > > > Hi there,<br>> > > > <br>> > > > I am a new user of syslog-ng. Thanks for your time and attention<br>> > in<br>> > > > advance.<br>> > > > <br>> > > > The problem is syslog-ng coredumps when use "internal()" in<br>> > "source",<br>> > > > however it will work fine without "internal()". The conf file is<br>> > like:<br>> > > <br>> > > <br>> > > hmm... g_malloc() never fails, it aborts() instead after logging the<br>> > > failure using assert.<br>> > > <br>> > > This assert is then caught by syslog-ng, to be put in the internal()<br>> > > message source , but then again allocation fails, thus the segfault.<br>> > > <br>> > > Are you sure that syslog-ng is not severely constrained by ulimit or<br>> > > something?<br>> > > <br>> > > Generally if memory allocation fails, syslog-ng aborts.<br>> > > <br>> > > Ah, as I see you said the core file is large, this seems to be some<br>> > kind<br>> > > of memory leak, or overuse of memory, and that's the root reason of<br>> > the<br>> > > crash.<br>> > > <br>> > > Can you send me your complete configuration file? Maybe we could<br>> > > reproduce the problem. Do you happen to have large log_fifo_size()<br>> > > configured to one of your destinations?<br>> > > <br>> > > > <br>> > > > ===>BEGIN<br>> > > > @version:3.0<br>> > > > <br>> > > > options { dir_perm(0755); perm(0644);<br>> > > > chain_hostnames(no);<br>> > > > keep_hostname(yes);<br>> > > > };<br>> > > > <br>> > > > # Default local source.<br>> > > > source local {<br>> > > > unix-dgram("/var/run/log");<br>> > > > unix-dgram("/var/run/logpriv" perm(0600));<br>> > > > internal();<br>> > > > file("/dev/klog");<br>> > > > };<br>> > > > <br>> > > > ...<br>> > > > <br>> > > > filter f_hecxd { program("hecxd"); };<br>> > > > <br>> > > > ...<br>> > > > <br>> > > > log { source(local); filter(f_hecxd); destination(d_hecxd); };<br>> > > > ==>END<br>> > > > <br>> > > > The options used for building syslog-ng is: <br>> > > > <br>> > > > CFLAGS='-Wl,-Bdynamic -liconv -Wl,-Bstatic'<br>> > --enable-static-linking<br>> > > > \<br>> > > > --enable-debug \<br>> > > > --enable-ssl --enable-ipv6=no --enable-tcp-wrapper=no \<br>> > > > --with-pidfile-dir="/var/run/"<br>> > > > <br>> > > > The platform is: FREEBSD 6.4-RELEASE-p6 i386<br>> > > > <br>> > > > I got a very big core dump file (>500M), by backtracing that file,<br>> > I<br>> > > > can see the problem is "g_malloc0" for some reason failed to<br>> > allocate<br>> > > > memory, while syslog-ng did not check the return value and tried<br>> > to<br>> > > > use the "0x0" pointer in a "strlen()" call later. Here is a part<br>> > of<br>> > > > trace information:<br>> > > > <br>> > > > #0 0x080a2d8d in strlen ()<br>> > > > No symbol table info available.<br>> > > > #1 0x0804e7fe in log_msg_new_internal (prio=43, msg=0x0, flags=6)<br>> > at<br>> > > > logmsg.c:1971<br>> > > > buf = (gchar *) 0x0<br>> > > > self = (LogMessage *) 0x8159000<br>> > > > #2 0x080492b1 in msg_send_internal_message (prio=43, msg=0x0) at<br>> > > > messages.c:128<br>> > > > context = (MsgContext *) 0x81893a0<br>> > > > m = (LogMessage *) 0x7<br>> > > > #3 0x0804954c in msg_log_func (log_domain=0x81186ea "GLib",<br>> > > > log_flags=6, msg=0x0, user_data=0x0) at messages.c:187<br>> > > > pri = 43<br>> > > > #4 0x08084906 in IA__g_logv (log_domain=0x81186ea "GLib",<br>> > > > log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to<br>> > allocate<br>> > > > %u bytes",<br>> > > > args1=0xbfbfe6cc "3�\021\bl") at gmessages.c:519<br>> > > > msg = (gchar *) 0x0<br>> > > > was_fatal = 0<br>> > > > was_recursion = 0<br>> > > > i = 2<br>> > > > #5 0x08084ae4 in IA__g_log (log_domain=0x81186ea "GLib",<br>> > > > log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to<br>> > allocate<br>> > > > %u bytes")<br>> > > > at gmessages.c:569<br>> > > > No locals.<br>> > > > #6 0x08084239 in IA__g_malloc0 (n_bytes=108) at gmem.c:155<br>> > > > mem = 0x0<br>> > > > #7 0x0804e5ef in log_msg_new (<br>> > > > msg=0x8a82180 "<47> syslog-ng[3277]: Filter node evaluation<br>> > > > result; filter_result='not-match', filter_type='level'\n",<br>> > length=100,<br>> > > > saddr=0x0, flags=6, bad_hostname=0x0, assume_timezone=-1,<br>> > > > default_pri=47) at logmsg.c:1873<br>> > > > self = (LogMessage *) 0x810ba1b<br>> > > > #8 0x0804e80a in log_msg_new_internal (prio=47,<br>> > > > msg=0x817f500 "Filter node evaluation result;<br>> > > > filter_result='not-match', filter_type='level'", flags=6) at<br>> > > > logmsg.c:1971<br>> > > > buf = (gchar *) 0x8a82180 "<47> syslog-ng[3277]: Filter node<br>> > > > evaluation result; filter_result='not-match',<br>> > filter_type='level'\n"<br>> > > > self = (LogMessage *) 0x8159000<br>> > > > #9 0x080492b1 in msg_send_internal_message (prio=47,<br>> > > > msg=0x817f500 "Filter node evaluation result;<br>> > > > filter_result='not-match', filter_type='level'") at messages.c:128<br>> > > > context = (MsgContext *) 0x81893a0<br>> > > > m = (LogMessage *) 0x8182030<br>> > > > <br>> > > > If you need the completed coredump file, just let me know and tell<br>> > me<br>> > > > where to put it :)<br>> > > > <br>> > > > Any help is much appreciated.<br>> > > > <br>> > > > Thanks a lot.<br>> > > > Shen<br>> > > > <br>> > > > <br>> > > > <br>> > > > <br>> > > ><br>> > ______________________________________________________________________<br>> > > > Windows Live: Make it easier for your friends to see what you’re<br>> > up to<br>> > > > on Facebook.<br>> > > ><br>> > ______________________________________________________________________________<br>> > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>> > > > Documentation:<br>> > http://www.balabit.com/support/documentation/?product=syslog-ng<br>> > > > FAQ: http://www.campin.net/syslog-ng/faq.html<br>> > > > <br>> > > <br>> > > <br>> > > -- <br>> > > Bazsi<br>> > > <br>> > > <br>> > ><br>> > ______________________________________________________________________________<br>> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>> > > Documentation:<br>> > http://www.balabit.com/support/documentation/?product=syslog-ng<br>> > > FAQ: http://www.campin.net/syslog-ng/faq.html<br>> > > <br>> > <br>> > <br>> > ______________________________________________________________________<br>> > Get a great deal on Windows 7 and see how it works the way you want.<br>> > Check out the offers on Windows 7now.<br>> > ______________________________________________________________________________<br>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<br>> > FAQ: http://www.campin.net/syslog-ng/faq.html<br>> > <br>> <br>> <br>> -- <br>> Bazsi<br>> <br>> ______________________________________________________________________________<br>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<br>> FAQ: http://www.campin.net/syslog-ng/faq.html<br>> <br> <br /><hr />Get a great deal on Windows 7 and see how it works the way you want. <a href='http://go.microsoft.com/?linkid=9691813' target='_new'>See the Windows 7 offers now.</a></body>
</html>