[syslog-ng] syslog-ng OSE 3.0.4 coredumps when use "internal()" in s_local

Balazs Scheidler bazsi at balabit.hu
Tue Dec 15 18:04:24 CET 2009 

On Fri, 2009-12-11 at 18:21 +0000, nova club wrote:
> Thank you so much for your attention. I tried 3.0.5 and it has same
> problem. Please see the attached tarball consisting of a conf file and
> static-linked syslog-ng binary.
> 
> The platform I am using is  "6.4-RELEASE-p6 FreeBSD 6.4-RELEASE-p6
> i386". The command line is like "syslog-ng -f <conf> -d -F". And for
> now, I have not found any special or abnormal resource "limit" setup
> on that box. 
> 
> Another interesting thing is no coredumps if I use "-e" option too.
> 
> The conf file actually is a copy of
> "contrib/freebsd-packaging/syslog-ng.conf.example", I just added a new
> log path for a program called "hecxd".
> 
> Thanks a lot again.

I don't have any more time, but the problem seems to be caused by the
fact that --debug messages are enabled.

filtering generates a 'debug' message, which in turn is processed by the
same filter generating another filter and so on.

So effectively syslog-ng is busy processing its own message. And these
messages are collected in memory (possibly in a destination queue), thus
syslog-ng is busy allocating memory too.

At the end of the day, the memory runs out, causing an abort, which will
cause a segfault.

There's explicit processing in syslog-ng to stop such message loops
caused by internal(), but this is clearly not triggering this case.

Disabling --debug messages should resolve your problem though.

> 
> > From: bazsi at balabit.hu
> > To: syslog-ng at lists.balabit.hu
> > Date: Fri, 11 Dec 2009 07:52:02 +0100
> > Subject: Re: [syslog-ng] syslog-ng OSE 3.0.4 coredumps when use
> "internal()" in s_local
> > 
> > On Wed, 2009-12-09 at 18:04 +0000, nova club wrote:
> > > Hi there,
> > > 
> > > I am a new user of syslog-ng. Thanks for your time and attention
> in
> > > advance.
> > > 
> > > The problem is syslog-ng coredumps when use "internal()" in
> "source",
> > > however it will work fine without "internal()". The conf file is
> like:
> > 
> > 
> > hmm... g_malloc() never fails, it aborts() instead after logging the
> > failure using assert.
> > 
> > This assert is then caught by syslog-ng, to be put in the internal()
> > message source , but then again allocation fails, thus the segfault.
> > 
> > Are you sure that syslog-ng is not severely constrained by ulimit or
> > something?
> > 
> > Generally if memory allocation fails, syslog-ng aborts.
> > 
> > Ah, as I see you said the core file is large, this seems to be some
> kind
> > of memory leak, or overuse of memory, and that's the root reason of
> the
> > crash.
> > 
> > Can you send me your complete configuration file? Maybe we could
> > reproduce the problem. Do you happen to have large log_fifo_size()
> > configured to one of your destinations?
> > 
> > > 
> > > ===>BEGIN
> > > @version:3.0
> > > 
> > > options { dir_perm(0755); perm(0644);
> > > chain_hostnames(no);
> > > keep_hostname(yes);
> > > };
> > > 
> > > # Default local source.
> > > source local {
> > > unix-dgram("/var/run/log");
> > > unix-dgram("/var/run/logpriv" perm(0600));
> > > internal();
> > > file("/dev/klog");
> > > };
> > > 
> > > ...
> > > 
> > > filter f_hecxd { program("hecxd"); };
> > > 
> > > ...
> > > 
> > > log { source(local); filter(f_hecxd); destination(d_hecxd); };
> > > ==>END
> > > 
> > > The options used for building syslog-ng is: 
> > > 
> > > CFLAGS='-Wl,-Bdynamic -liconv -Wl,-Bstatic'
> --enable-static-linking
> > > \
> > > --enable-debug \
> > > --enable-ssl --enable-ipv6=no --enable-tcp-wrapper=no \
> > > --with-pidfile-dir="/var/run/"
> > > 
> > > The platform is: FREEBSD 6.4-RELEASE-p6 i386
> > > 
> > > I got a very big core dump file (>500M), by backtracing that file,
> I
> > > can see the problem is "g_malloc0" for some reason failed to
> allocate
> > > memory, while syslog-ng did not check the return value and tried
> to
> > > use the "0x0" pointer in a "strlen()" call later. Here is a part
> of
> > > trace information:
> > > 
> > > #0 0x080a2d8d in strlen ()
> > > No symbol table info available.
> > > #1 0x0804e7fe in log_msg_new_internal (prio=43, msg=0x0, flags=6)
> at
> > > logmsg.c:1971
> > > buf = (gchar *) 0x0
> > > self = (LogMessage *) 0x8159000
> > > #2 0x080492b1 in msg_send_internal_message (prio=43, msg=0x0) at
> > > messages.c:128
> > > context = (MsgContext *) 0x81893a0
> > > m = (LogMessage *) 0x7
> > > #3 0x0804954c in msg_log_func (log_domain=0x81186ea "GLib",
> > > log_flags=6, msg=0x0, user_data=0x0) at messages.c:187
> > > pri = 43
> > > #4 0x08084906 in IA__g_logv (log_domain=0x81186ea "GLib",
> > > log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to
> allocate
> > > %u bytes",
> > > args1=0xbfbfe6cc "3�\021\bl") at gmessages.c:519
> > > msg = (gchar *) 0x0
> > > was_fatal = 0
> > > was_recursion = 0
> > > i = 2
> > > #5 0x08084ae4 in IA__g_log (log_domain=0x81186ea "GLib",
> > > log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to
> allocate
> > > %u bytes")
> > > at gmessages.c:569
> > > No locals.
> > > #6 0x08084239 in IA__g_malloc0 (n_bytes=108) at gmem.c:155
> > > mem = 0x0
> > > #7 0x0804e5ef in log_msg_new (
> > > msg=0x8a82180 "<47> syslog-ng[3277]: Filter node evaluation
> > > result; filter_result='not-match', filter_type='level'\n",
> length=100,
> > > saddr=0x0, flags=6, bad_hostname=0x0, assume_timezone=-1,
> > > default_pri=47) at logmsg.c:1873
> > > self = (LogMessage *) 0x810ba1b
> > > #8 0x0804e80a in log_msg_new_internal (prio=47,
> > > msg=0x817f500 "Filter node evaluation result;
> > > filter_result='not-match', filter_type='level'", flags=6) at
> > > logmsg.c:1971
> > > buf = (gchar *) 0x8a82180 "<47> syslog-ng[3277]: Filter node
> > > evaluation result; filter_result='not-match',
> filter_type='level'\n"
> > > self = (LogMessage *) 0x8159000
> > > #9 0x080492b1 in msg_send_internal_message (prio=47,
> > > msg=0x817f500 "Filter node evaluation result;
> > > filter_result='not-match', filter_type='level'") at messages.c:128
> > > context = (MsgContext *) 0x81893a0
> > > m = (LogMessage *) 0x8182030
> > > 
> > > If you need the completed coredump file, just let me know and tell
> me
> > > where to put it :)
> > > 
> > > Any help is much appreciated.
> > > 
> > > Thanks a lot.
> > > Shen
> > > 
> > > 
> > > 
> > > 
> > >
> ______________________________________________________________________
> > > Windows Live: Make it easier for your friends to see what you’re
> up to
> > > on Facebook.
> > >
> ______________________________________________________________________________
> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > > FAQ: http://www.campin.net/syslog-ng/faq.html
> > > 
> > 
> > 
> > -- 
> > Bazsi
> > 
> > 
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.campin.net/syslog-ng/faq.html
> > 
> 
> 
> ______________________________________________________________________
> Get a great deal on Windows 7 and see how it works the way you want.
> Check out the offers on Windows 7now.
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 


-- 
Bazsi

More information about the syslog-ng mailing list