[syslog-ng] syslog-ng OSE 3.0.4 coredumps when use "internal()" in s_local

nova club clubnova22 at hotmail.com
Tue Dec 15 20:20:31 CET 2009 

Yes, you are absolutely right. I can see the infinite loop (with brk() call inside) from the tracing file.  Thank you so much for your great help which saved me a lot of time. I was thinking something wrong when I built glib :)

> To: syslog-ng at lists.balabit.hu
> Date: Tue, 15 Dec 2009 18:04:24 +0100
> Subject: Re: [syslog-ng] syslog-ng OSE 3.0.4 coredumps when use "internal()" in s_local
> 
> On Fri, 2009-12-11 at 18:21 +0000, nova club wrote:
> > Thank you so much for your attention. I tried 3.0.5 and it has same
> > problem. Please see the attached tarball consisting of a conf file and
> > static-linked syslog-ng binary.
> > 
> > The platform I am using is  "6.4-RELEASE-p6 FreeBSD 6.4-RELEASE-p6
> > i386". The command line is like "syslog-ng -f <conf> -d -F". And for
> > now, I have not found any special or abnormal resource "limit" setup
> > on that box. 
> > 
> > Another interesting thing is no coredumps if I use "-e" option too.
> > 
> > The conf file actually is a copy of
> > "contrib/freebsd-packaging/syslog-ng.conf.example", I just added a new
> > log path for a program called "hecxd".
> > 
> > Thanks a lot again.
> 
> I don't have any more time, but the problem seems to be caused by the
> fact that --debug messages are enabled.
> 
> filtering generates a 'debug' message, which in turn is processed by the
> same filter generating another filter and so on.
> 
> So effectively syslog-ng is busy processing its own message. And these
> messages are collected in memory (possibly in a destination queue), thus
> syslog-ng is busy allocating memory too.
> 
> At the end of the day, the memory runs out, causing an abort, which will
> cause a segfault.
> 
> There's explicit processing in syslog-ng to stop such message loops
> caused by internal(), but this is clearly not triggering this case.
> 
> Disabling --debug messages should resolve your problem though.
> 
> > 
> > > From: bazsi at balabit.hu
> > > To: syslog-ng at lists.balabit.hu
> > > Date: Fri, 11 Dec 2009 07:52:02 +0100
> > > Subject: Re: [syslog-ng] syslog-ng OSE 3.0.4 coredumps when use
> > "internal()" in s_local
> > > 
> > > On Wed, 2009-12-09 at 18:04 +0000, nova club wrote:
> > > > Hi there,
> > > > 
> > > > I am a new user of syslog-ng. Thanks for your time and attention
> > in
> > > > advance.
> > > > 
> > > > The problem is syslog-ng coredumps when use "internal()" in
> > "source",
> > > > however it will work fine without "internal()". The conf file is
> > like:
> > > 
> > > 
> > > hmm... g_malloc() never fails, it aborts() instead after logging the
> > > failure using assert.
> > > 
> > > This assert is then caught by syslog-ng, to be put in the internal()
> > > message source , but then again allocation fails, thus the segfault.
> > > 
> > > Are you sure that syslog-ng is not severely constrained by ulimit or
> > > something?
> > > 
> > > Generally if memory allocation fails, syslog-ng aborts.
> > > 
> > > Ah, as I see you said the core file is large, this seems to be some
> > kind
> > > of memory leak, or overuse of memory, and that's the root reason of
> > the
> > > crash.
> > > 
> > > Can you send me your complete configuration file? Maybe we could
> > > reproduce the problem. Do you happen to have large log_fifo_size()
> > > configured to one of your destinations?
> > > 
> > > > 
> > > > ===>BEGIN
> > > > @version:3.0
> > > > 
> > > > options { dir_perm(0755); perm(0644);
> > > > chain_hostnames(no);
> > > > keep_hostname(yes);
> > > > };
> > > > 
> > > > # Default local source.
> > > > source local {
> > > > unix-dgram("/var/run/log");
> > > > unix-dgram("/var/run/logpriv" perm(0600));
> > > > internal();
> > > > file("/dev/klog");
> > > > };
> > > > 
> > > > ...
> > > > 
> > > > filter f_hecxd { program("hecxd"); };
> > > > 
> > > > ...
> > > > 
> > > > log { source(local); filter(f_hecxd); destination(d_hecxd); };
> > > > ==>END
> > > > 
> > > > The options used for building syslog-ng is: 
> > > > 
> > > > CFLAGS='-Wl,-Bdynamic -liconv -Wl,-Bstatic'
> > --enable-static-linking
> > > > \
> > > > --enable-debug \
> > > > --enable-ssl --enable-ipv6=no --enable-tcp-wrapper=no \
> > > > --with-pidfile-dir="/var/run/"
> > > > 
> > > > The platform is: FREEBSD 6.4-RELEASE-p6 i386
> > > > 
> > > > I got a very big core dump file (>500M), by backtracing that file,
> > I
> > > > can see the problem is "g_malloc0" for some reason failed to
> > allocate
> > > > memory, while syslog-ng did not check the return value and tried
> > to
> > > > use the "0x0" pointer in a "strlen()" call later. Here is a part
> > of
> > > > trace information:
> > > > 
> > > > #0 0x080a2d8d in strlen ()
> > > > No symbol table info available.
> > > > #1 0x0804e7fe in log_msg_new_internal (prio=43, msg=0x0, flags=6)
> > at
> > > > logmsg.c:1971
> > > > buf = (gchar *) 0x0
> > > > self = (LogMessage *) 0x8159000
> > > > #2 0x080492b1 in msg_send_internal_message (prio=43, msg=0x0) at
> > > > messages.c:128
> > > > context = (MsgContext *) 0x81893a0
> > > > m = (LogMessage *) 0x7
> > > > #3 0x0804954c in msg_log_func (log_domain=0x81186ea "GLib",
> > > > log_flags=6, msg=0x0, user_data=0x0) at messages.c:187
> > > > pri = 43
> > > > #4 0x08084906 in IA__g_logv (log_domain=0x81186ea "GLib",
> > > > log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to
> > allocate
> > > > %u bytes",
> > > > args1=0xbfbfe6cc "3�\021\bl") at gmessages.c:519
> > > > msg = (gchar *) 0x0
> > > > was_fatal = 0
> > > > was_recursion = 0
> > > > i = 2
> > > > #5 0x08084ae4 in IA__g_log (log_domain=0x81186ea "GLib",
> > > > log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to
> > allocate
> > > > %u bytes")
> > > > at gmessages.c:569
> > > > No locals.
> > > > #6 0x08084239 in IA__g_malloc0 (n_bytes=108) at gmem.c:155
> > > > mem = 0x0
> > > > #7 0x0804e5ef in log_msg_new (
> > > > msg=0x8a82180 "<47> syslog-ng[3277]: Filter node evaluation
> > > > result; filter_result='not-match', filter_type='level'\n",
> > length=100,
> > > > saddr=0x0, flags=6, bad_hostname=0x0, assume_timezone=-1,
> > > > default_pri=47) at logmsg.c:1873
> > > > self = (LogMessage *) 0x810ba1b
> > > > #8 0x0804e80a in log_msg_new_internal (prio=47,
> > > > msg=0x817f500 "Filter node evaluation result;
> > > > filter_result='not-match', filter_type='level'", flags=6) at
> > > > logmsg.c:1971
> > > > buf = (gchar *) 0x8a82180 "<47> syslog-ng[3277]: Filter node
> > > > evaluation result; filter_result='not-match',
> > filter_type='level'\n"
> > > > self = (LogMessage *) 0x8159000
> > > > #9 0x080492b1 in msg_send_internal_message (prio=47,
> > > > msg=0x817f500 "Filter node evaluation result;
> > > > filter_result='not-match', filter_type='level'") at messages.c:128
> > > > context = (MsgContext *) 0x81893a0
> > > > m = (LogMessage *) 0x8182030
> > > > 
> > > > If you need the completed coredump file, just let me know and tell
> > me
> > > > where to put it :)
> > > > 
> > > > Any help is much appreciated.
> > > > 
> > > > Thanks a lot.
> > > > Shen
> > > > 
> > > > 
> > > > 
> > > > 
> > > >
> > ______________________________________________________________________
> > > > Windows Live: Make it easier for your friends to see what you’re
> > up to
> > > > on Facebook.
> > > >
> > ______________________________________________________________________________
> > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > > > FAQ: http://www.campin.net/syslog-ng/faq.html
> > > > 
> > > 
> > > 
> > > -- 
> > > Bazsi
> > > 
> > > 
> > >
> > ______________________________________________________________________________
> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > > FAQ: http://www.campin.net/syslog-ng/faq.html
> > > 
> > 
> > 
> > ______________________________________________________________________
> > Get a great deal on Windows 7 and see how it works the way you want.
> > Check out the offers on Windows 7now.
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.campin.net/syslog-ng/faq.html
> > 
> 
> 
> -- 
> Bazsi
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 
 		 	   		  
_________________________________________________________________
Ready. Set. Get a great deal on Windows 7. See fantastic deals on Windows 7 now
http://go.microsoft.com/?linkid=9691818
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20091215/928e43d6/attachment.htm

More information about the syslog-ng mailing list