[syslog-ng] Ongoing 3.3.4 memory leaks

[Marvin Nipper]

Thanks for the quick reply.  See my comments below.

> -----Original Message-----
> From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-
> bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
> Sent: Saturday, March 31, 2012 2:22 PM
> To: Syslog-ng users' and developers' mailing list
> Subject: Re: [syslog-ng] Ongoing 3.3.4 memory leaks
> 
> On Fri, 2012-03-30 at 16:52 -0500, Marvin Nipper wrote:
> > Hi.
> >
> > Sorry, looking for some help.  Still trying to chase some memory
> leaks
> > that I see on Solaris 10.  I’ll crank up syslog-ng, and it’ll start
> > running at around 7 or 8MB, and interestingly, it will run that way
> > sometimes for several hours, and then (for reasons unknown), I’ll
> take
> > a look at it a bit later, and suddenly it will have jumped to 26MB
> (or
> > later on, maybe 75MB).  I’ve already applied the (older) logwriter.c
> > fix and the (new) Gergely driver.h/afsocket.c fix, in an attempt to
> > resolve known issues.  Certainly the logwriter fix corrected a
> > super-nasty, continuous leak problem, but this other leak problem
> > still lingers.
> 
> I'm not sure which fixes you've applied. Can you post a better
> reference?
[Marvin Nipper] 
They are the logwriter fix that you mentioned just below, and also the afsocket patch that you were discussing this weekend in the "Memory leak reintroduced in 3.3.3" thread.  However, based upon your comments (in that thread), I plan to "unwind" that set of changes, as it seems like those may not have been legit.



> 
> I've applied only one patch after v3.3.4 got released which fixes a
> memory leak, this one:
> 
>     logwriter: fixed a memory leak for messages that couldn't be
> flushed at once
> 
>     Sometimes a message is dequeued, and then can't be posted to the
> destination
>     (probably because of an EAGAIN or something like that). In this
> case
>     it is readded back to the queue. In this case a reference was lost,
> causing
>     a memory leak, but not for all messages, but only for the affected
> ones.
> 
>     Reported-By: Evan Rempel <erempel at uvic.ca>
>     Signed-off-by: Balazs Scheidler <bazsi at balabit.hu>
> 
> I can't remember anything else, hopefully I haven't missed patches by
> Gergely.
> >
> > MY PROBLEM is that I’m wanting to use the debug symbols in syslog-ng,
> > so that I can have the Solaris tools specifically show the exact
> > location where these leaks are happening, BUT, whereas a non-
> debugging
> > version of syslog-ng executes properly, when I configure syslog-ng
> > with –enable-debug, the resulting executable will no longer start.
> It
> > immediately exits, coughing up what amounts to a syntax error (even
> > though I’m using the exact same command-line invocation that I use on
> > the non-debugged executable):
> >
> >
> > #/etc/init.d/syslog-ng start
> > syslog-ng service starting.
> > mkfifo: File exists
> > mkfifo: File exists
> > Usage: /usr/local/sbin/syslog-ng { start | stop }
> > #
> >
> > Something about enabling debugging is impacting the operational
> > behavior of the resulting binary.
> 
> Hmm.  That output is strange, you said you were running Solaris, but
> Solaris uses SMF for service management. Yet, you executed an sysv init
> script, which then complained that /usr/local/sbin/syslog-ng has
> different options, even though /usr/local/sbin/syslog-ng is probably
> your syslog-ng executable (or at least that's the default path).
> 
> Can you perhaps start syslog-ng by hand?
> 
> /usr/local/sbin/syslog-ng -F
> 
> should do it.
[Marvin Nipper] 
Actually, I had already done that, but here's that output, just to show you:
# /usr/local/sbin/syslog-ng \
>   -F -f /etc/syslog-ng/syslog-ng.conf \
>          -p /var/run/syslog-ng.pid \
>          --persist-file=/usr/local/syslog-ng/syslog-ng.persist
Usage: /usr/local/sbin/syslog-ng { start | stop }




 Debug symbols shouldn't affect how things run.
> --enable-debug only adds a couple of options to CFLAGS, and only if the
> compiler is gcc.
> 
> if test "x$ac_compiler_gnu" = "xyes"; then
>         CFLAGS="${CFLAGS} -Wall"
>         if test "x$enable_debug" = "xyes"; then
>                 CFLAGS="${ac_cv_env_CFLAGS_value} -Wall -g"
>         fi
> 
>         if test "x$enable_gprof" = "xyes"; then
>                 CFLAGS="${CFLAGS} -pg"
>         fi
>         if test "x$enable_gcov" = "xyes"; then
>                 CFLAGS="${CFLAGS} -fprofile-arcs -ftest-coverage"
>         fi
> fi
> CFLAGS="${CFLAGS} -pthread"
> 
> If nothing else, try passing -g in CFLAGS before running configure,
> without
> passing --enable-debug.
[Marvin Nipper] 
I actually didn't even know that the --enable-debug option existed in the syslog-ng config, and so I started this effort by doing a compilation with -g in CFLAGS, and that's when I first encountered the syntax issue (that results in the "Usage" error).  When that goofy result occurred, I figured that I'd done something wrong, and then stumbled into the debug capability "built into" your configure scripting, so I tried that next, which ends up with the same result.

So, I don't know if this is yet another "goofy Solaris problem", but some aspect of enabling the debug symbols, seems to frak up some aspect of processing the command line at startup.

If you have any other ideas or thoughts, I'll certainly take them.  I appreciate your time and input.



> 
> >
> > Anyway, I can use (for Solaris) libumem and mdb to narrow this down
> to
> > a particular module, but without the debug symbols, I can’t really
> get
> > to a specific chunk of code, so it would obviously help to get that
> > working.  If anyone has any ideas, I certainly would appreciate the
> > input.
> >
> 
> It'd really be appreciated if you could point out where the leak is
> happening. It might be use-case specific or some failure handler or
> whatever. Let's hope that starting syslog-ng from the command line
> works.
> 
> --
> Bazsi
> 
> 
> _______________________________________________________________________
> _______
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq



This e-mail may contain confidential and/or privileged information. If you are
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy this e-mail. Any unauthorized 
copying, disclosure or distribution of the material in this e-mail is strictly 
forbidden.