[syslog-ng] Ongoing 3.3.4 memory leaks

[Balazs Scheidler]

On Fri, 2012-03-30 at 16:52 -0500, Marvin Nipper wrote:
> Hi.
>  
> Sorry, looking for some help.  Still trying to chase some memory leaks
> that I see on Solaris 10.  I’ll crank up syslog-ng, and it’ll start
> running at around 7 or 8MB, and interestingly, it will run that way
> sometimes for several hours, and then (for reasons unknown), I’ll take
> a look at it a bit later, and suddenly it will have jumped to 26MB (or
> later on, maybe 75MB).  I’ve already applied the (older) logwriter.c
> fix and the (new) Gergely driver.h/afsocket.c fix, in an attempt to
> resolve known issues.  Certainly the logwriter fix corrected a
> super-nasty, continuous leak problem, but this other leak problem
> still lingers.

I'm not sure which fixes you've applied. Can you post a better
reference?

I've applied only one patch after v3.3.4 got released which fixes a
memory leak, this one:

    logwriter: fixed a memory leak for messages that couldn't be flushed at once
    
    Sometimes a message is dequeued, and then can't be posted to the destination
    (probably because of an EAGAIN or something like that). In this case
    it is readded back to the queue. In this case a reference was lost, causing
    a memory leak, but not for all messages, but only for the affected ones.
    
    Reported-By: Evan Rempel <erempel at uvic.ca>
    Signed-off-by: Balazs Scheidler <bazsi at balabit.hu>

I can't remember anything else, hopefully I haven't missed patches by Gergely.
> 
> MY PROBLEM is that I’m wanting to use the debug symbols in syslog-ng,
> so that I can have the Solaris tools specifically show the exact
> location where these leaks are happening, BUT, whereas a non-debugging
> version of syslog-ng executes properly, when I configure syslog-ng
> with –enable-debug, the resulting executable will no longer start.  It
> immediately exits, coughing up what amounts to a syntax error (even
> though I’m using the exact same command-line invocation that I use on
> the non-debugged executable):
>  
>  
> #/etc/init.d/syslog-ng start
> syslog-ng service starting.
> mkfifo: File exists
> mkfifo: File exists
> Usage: /usr/local/sbin/syslog-ng { start | stop }
> #
>  
> Something about enabling debugging is impacting the operational
> behavior of the resulting binary.

Hmm.  That output is strange, you said you were running Solaris, but
Solaris uses SMF for service management. Yet, you executed an sysv init
script, which then complained that /usr/local/sbin/syslog-ng has
different options, even though /usr/local/sbin/syslog-ng is probably
your syslog-ng executable (or at least that's the default path).

Can you perhaps start syslog-ng by hand?

/usr/local/sbin/syslog-ng -F

should do it. Debug symbols shouldn't affect how things run.
--enable-debug only adds a couple of options to CFLAGS, and only if the
compiler is gcc.

if test "x$ac_compiler_gnu" = "xyes"; then
        CFLAGS="${CFLAGS} -Wall"
        if test "x$enable_debug" = "xyes"; then
                CFLAGS="${ac_cv_env_CFLAGS_value} -Wall -g"
        fi

        if test "x$enable_gprof" = "xyes"; then
                CFLAGS="${CFLAGS} -pg"
        fi
        if test "x$enable_gcov" = "xyes"; then
                CFLAGS="${CFLAGS} -fprofile-arcs -ftest-coverage"
        fi
fi
CFLAGS="${CFLAGS} -pthread"

If nothing else, try passing -g in CFLAGS before running configure, without 
passing --enable-debug.

>  
> Anyway, I can use (for Solaris) libumem and mdb to narrow this down to
> a particular module, but without the debug symbols, I can’t really get
> to a specific chunk of code, so it would obviously help to get that
> working.  If anyone has any ideas, I certainly would appreciate the
> input.
>  

It'd really be appreciated if you could point out where the leak is
happening. It might be use-case specific or some failure handler or
whatever. Let's hope that starting syslog-ng from the command line
works.

-- 
Bazsi