Ongoing 3.3.4 memory leaks
Hi. Sorry, looking for some help. Still trying to chase some memory leaks that I see on Solaris 10. I'll crank up syslog-ng, and it'll start running at around 7 or 8MB, and interestingly, it will run that way sometimes for several hours, and then (for reasons unknown), I'll take a look at it a bit later, and suddenly it will have jumped to 26MB (or later on, maybe 75MB). I've already applied the (older) logwriter.c fix and the (new) Gergely driver.h/afsocket.c fix, in an attempt to resolve known issues. Certainly the logwriter fix corrected a super-nasty, continuous leak problem, but this other leak problem still lingers. MY PROBLEM is that I'm wanting to use the debug symbols in syslog-ng, so that I can have the Solaris tools specifically show the exact location where these leaks are happening, BUT, whereas a non-debugging version of syslog-ng executes properly, when I configure syslog-ng with -enable-debug, the resulting executable will no longer start. It immediately exits, coughing up what amounts to a syntax error (even though I'm using the exact same command-line invocation that I use on the non-debugged executable): #/etc/init.d/syslog-ng start syslog-ng service starting. mkfifo: File exists mkfifo: File exists Usage: /usr/local/sbin/syslog-ng { start | stop } # Something about enabling debugging is impacting the operational behavior of the resulting binary. Anyway, I can use (for Solaris) libumem and mdb to narrow this down to a particular module, but without the debug symbols, I can't really get to a specific chunk of code, so it would obviously help to get that working. If anyone has any ideas, I certainly would appreciate the input. THANKS Marvin Nipper This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
On Fri, 2012-03-30 at 16:52 -0500, Marvin Nipper wrote:
Hi.
Sorry, looking for some help. Still trying to chase some memory leaks that I see on Solaris 10. I’ll crank up syslog-ng, and it’ll start running at around 7 or 8MB, and interestingly, it will run that way sometimes for several hours, and then (for reasons unknown), I’ll take a look at it a bit later, and suddenly it will have jumped to 26MB (or later on, maybe 75MB). I’ve already applied the (older) logwriter.c fix and the (new) Gergely driver.h/afsocket.c fix, in an attempt to resolve known issues. Certainly the logwriter fix corrected a super-nasty, continuous leak problem, but this other leak problem still lingers.
I'm not sure which fixes you've applied. Can you post a better reference? I've applied only one patch after v3.3.4 got released which fixes a memory leak, this one: logwriter: fixed a memory leak for messages that couldn't be flushed at once Sometimes a message is dequeued, and then can't be posted to the destination (probably because of an EAGAIN or something like that). In this case it is readded back to the queue. In this case a reference was lost, causing a memory leak, but not for all messages, but only for the affected ones. Reported-By: Evan Rempel <erempel@uvic.ca> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> I can't remember anything else, hopefully I haven't missed patches by Gergely.
MY PROBLEM is that I’m wanting to use the debug symbols in syslog-ng, so that I can have the Solaris tools specifically show the exact location where these leaks are happening, BUT, whereas a non-debugging version of syslog-ng executes properly, when I configure syslog-ng with –enable-debug, the resulting executable will no longer start. It immediately exits, coughing up what amounts to a syntax error (even though I’m using the exact same command-line invocation that I use on the non-debugged executable):
#/etc/init.d/syslog-ng start syslog-ng service starting. mkfifo: File exists mkfifo: File exists Usage: /usr/local/sbin/syslog-ng { start | stop } #
Something about enabling debugging is impacting the operational behavior of the resulting binary.
Hmm. That output is strange, you said you were running Solaris, but Solaris uses SMF for service management. Yet, you executed an sysv init script, which then complained that /usr/local/sbin/syslog-ng has different options, even though /usr/local/sbin/syslog-ng is probably your syslog-ng executable (or at least that's the default path). Can you perhaps start syslog-ng by hand? /usr/local/sbin/syslog-ng -F should do it. Debug symbols shouldn't affect how things run. --enable-debug only adds a couple of options to CFLAGS, and only if the compiler is gcc. if test "x$ac_compiler_gnu" = "xyes"; then CFLAGS="${CFLAGS} -Wall" if test "x$enable_debug" = "xyes"; then CFLAGS="${ac_cv_env_CFLAGS_value} -Wall -g" fi if test "x$enable_gprof" = "xyes"; then CFLAGS="${CFLAGS} -pg" fi if test "x$enable_gcov" = "xyes"; then CFLAGS="${CFLAGS} -fprofile-arcs -ftest-coverage" fi fi CFLAGS="${CFLAGS} -pthread" If nothing else, try passing -g in CFLAGS before running configure, without passing --enable-debug.
Anyway, I can use (for Solaris) libumem and mdb to narrow this down to a particular module, but without the debug symbols, I can’t really get to a specific chunk of code, so it would obviously help to get that working. If anyone has any ideas, I certainly would appreciate the input.
It'd really be appreciated if you could point out where the leak is happening. It might be use-case specific or some failure handler or whatever. Let's hope that starting syslog-ng from the command line works. -- Bazsi
Thanks for the quick reply. See my comments below.
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng- bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Saturday, March 31, 2012 2:22 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Ongoing 3.3.4 memory leaks
On Fri, 2012-03-30 at 16:52 -0500, Marvin Nipper wrote:
Hi.
Sorry, looking for some help. Still trying to chase some memory leaks that I see on Solaris 10. I’ll crank up syslog-ng, and it’ll start running at around 7 or 8MB, and interestingly, it will run that way sometimes for several hours, and then (for reasons unknown), I’ll take a look at it a bit later, and suddenly it will have jumped to 26MB (or later on, maybe 75MB). I’ve already applied the (older) logwriter.c fix and the (new) Gergely driver.h/afsocket.c fix, in an attempt to resolve known issues. Certainly the logwriter fix corrected a super-nasty, continuous leak problem, but this other leak problem still lingers.
I'm not sure which fixes you've applied. Can you post a better reference? [Marvin Nipper] They are the logwriter fix that you mentioned just below, and also the afsocket patch that you were discussing this weekend in the "Memory leak reintroduced in 3.3.3" thread. However, based upon your comments (in that thread), I plan to "unwind" that set of changes, as it seems like those may not have been legit.
I've applied only one patch after v3.3.4 got released which fixes a memory leak, this one:
logwriter: fixed a memory leak for messages that couldn't be flushed at once
Sometimes a message is dequeued, and then can't be posted to the destination (probably because of an EAGAIN or something like that). In this case it is readded back to the queue. In this case a reference was lost, causing a memory leak, but not for all messages, but only for the affected ones.
Reported-By: Evan Rempel <erempel@uvic.ca> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
I can't remember anything else, hopefully I haven't missed patches by Gergely.
MY PROBLEM is that I’m wanting to use the debug symbols in syslog-ng, so that I can have the Solaris tools specifically show the exact location where these leaks are happening, BUT, whereas a non-
debugging
version of syslog-ng executes properly, when I configure syslog-ng with –enable-debug, the resulting executable will no longer start. It immediately exits, coughing up what amounts to a syntax error (even though I’m using the exact same command-line invocation that I use on the non-debugged executable):
#/etc/init.d/syslog-ng start syslog-ng service starting. mkfifo: File exists mkfifo: File exists Usage: /usr/local/sbin/syslog-ng { start | stop } #
Something about enabling debugging is impacting the operational behavior of the resulting binary.
Hmm. That output is strange, you said you were running Solaris, but Solaris uses SMF for service management. Yet, you executed an sysv init script, which then complained that /usr/local/sbin/syslog-ng has different options, even though /usr/local/sbin/syslog-ng is probably your syslog-ng executable (or at least that's the default path).
Can you perhaps start syslog-ng by hand?
/usr/local/sbin/syslog-ng -F
should do it.
[Marvin Nipper] Actually, I had already done that, but here's that output, just to show you: # /usr/local/sbin/syslog-ng \
-F -f /etc/syslog-ng/syslog-ng.conf \ -p /var/run/syslog-ng.pid \ --persist-file=/usr/local/syslog-ng/syslog-ng.persist Usage: /usr/local/sbin/syslog-ng { start | stop }
Debug symbols shouldn't affect how things run.
--enable-debug only adds a couple of options to CFLAGS, and only if the compiler is gcc.
if test "x$ac_compiler_gnu" = "xyes"; then CFLAGS="${CFLAGS} -Wall" if test "x$enable_debug" = "xyes"; then CFLAGS="${ac_cv_env_CFLAGS_value} -Wall -g" fi
if test "x$enable_gprof" = "xyes"; then CFLAGS="${CFLAGS} -pg" fi if test "x$enable_gcov" = "xyes"; then CFLAGS="${CFLAGS} -fprofile-arcs -ftest-coverage" fi fi CFLAGS="${CFLAGS} -pthread"
If nothing else, try passing -g in CFLAGS before running configure, without passing --enable-debug. [Marvin Nipper] I actually didn't even know that the --enable-debug option existed in the syslog-ng config, and so I started this effort by doing a compilation with -g in CFLAGS, and that's when I first encountered the syntax issue (that results in the "Usage" error). When that goofy result occurred, I figured that I'd done something wrong, and then stumbled into the debug capability "built into" your configure scripting, so I tried that next, which ends up with the same result.
So, I don't know if this is yet another "goofy Solaris problem", but some aspect of enabling the debug symbols, seems to frak up some aspect of processing the command line at startup. If you have any other ideas or thoughts, I'll certainly take them. I appreciate your time and input.
Anyway, I can use (for Solaris) libumem and mdb to narrow this down
to
a particular module, but without the debug symbols, I can’t really get to a specific chunk of code, so it would obviously help to get that working. If anyone has any ideas, I certainly would appreciate the input.
It'd really be appreciated if you could point out where the leak is happening. It might be use-case specific or some failure handler or whatever. Let's hope that starting syslog-ng from the command line works.
-- Bazsi
_______________________________________________________________________ _______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
----- Original message -----
Thanks for the quick reply. See my comments below.
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng- bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Saturday, March 31, 2012 2:22 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Ongoing 3.3.4 memory leaks
On Fri, 2012-03-30 at 16:52 -0500, Marvin Nipper wrote:
Hi.
Sorry, looking for some help. Still trying to chase some memory leaks that I see on Solaris 10. I’ll crank up syslog-ng, and it’ll start running at around 7 or 8MB, and interestingly, it will run that way sometimes for several hours, and then (for reasons unknown), I’ll take a look at it a bit later, and suddenly it will have jumped to 26MB (or later on, maybe 75MB). I’ve already applied the (older) logwriter.c fix and the (new) Gergely driver.h/afsocket.c fix, in an attempt to resolve known issues. Certainly the logwriter fix corrected a super-nasty, continuous leak problem, but this other leak problem still lingers.
I'm not sure which fixes you've applied. Can you post a better reference? [Marvin Nipper] They are the logwriter fix that you mentioned just below, and also the afsocket patch that you were discussing this weekend in the "Memory leak reintroduced in 3.3.3" thread. However, based upon your comments (in that thread), I plan to "unwind" that set of changes, as it seems like those may not have been legit.
I've processed those after answering your email.
I've applied only one patch after v3.3.4 got released which fixes a memory leak, this one:
logwriter: fixed a memory leak for messages that couldn't be flushed at once
Sometimes a message is dequeued, and then can't be posted to the destination (probably because of an EAGAIN or something like that). In this case it is readded back to the queue. In this case a reference was lost, causing a memory leak, but not for all messages, but only for the affected ones.
Reported-By: Evan Rempel <erempel@uvic.ca> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
I can't remember anything else, hopefully I haven't missed patches by Gergely.
MY PROBLEM is that I’m wanting to use the debug symbols in syslog-ng, so that I can have the Solaris tools specifically show the exact location where these leaks are happening, BUT, whereas a non-
debugging
version of syslog-ng executes properly, when I configure syslog-ng with –enable-debug, the resulting executable will no longer start. It immediately exits, coughing up what amounts to a syntax error (even though I’m using the exact same command-line invocation that I use on the non-debugged executable):
#/etc/init.d/syslog-ng start syslog-ng service starting. mkfifo: File exists mkfifo: File exists Usage: /usr/local/sbin/syslog-ng { start | stop } #
Something about enabling debugging is impacting the operational behavior of the resulting binary.
Hmm. That output is strange, you said you were running Solaris, but Solaris uses SMF for service management. Yet, you executed an sysv init script, which then complained that /usr/local/sbin/syslog-ng has different options, even though /usr/local/sbin/syslog-ng is probably your syslog-ng executable (or at least that's the default path).
Can you perhaps start syslog-ng by hand?
/usr/local/sbin/syslog-ng -F
should do it.
[Marvin Nipper] Actually, I had already done that, but here's that output, just to show you: # /usr/local/sbin/syslog-ng \
-F -f /etc/syslog-ng/syslog-ng.conf \ -p /var/run/syslog-ng.pid \ --persist-file=/usr/local/syslog-ng/syslog-ng.persist Usage: /usr/local/sbin/syslog-ng { start | stop }
this isn't syslog-ng that produces this output. maybe the installation overwrote it? is that file a shell script?
Debug symbols shouldn't affect how things run.
--enable-debug only adds a couple of options to CFLAGS, and only if the compiler is gcc.
if test "x$ac_compiler_gnu" = "xyes"; then CFLAGS="${CFLAGS} -Wall" if test "x$enable_debug" = "xyes"; then CFLAGS="${ac_cv_env_CFLAGS_value} -Wall -g" fi
if test "x$enable_gprof" = "xyes"; then CFLAGS="${CFLAGS} -pg" fi if test "x$enable_gcov" = "xyes"; then CFLAGS="${CFLAGS} -fprofile-arcs -ftest-coverage" fi fi CFLAGS="${CFLAGS} -pthread"
If nothing else, try passing -g in CFLAGS before running configure, without passing --enable-debug. [Marvin Nipper] I actually didn't even know that the --enable-debug option existed in the syslog-ng config, and so I started this effort by doing a compilation with -g in CFLAGS, and that's when I first encountered the syntax issue (that results in the "Usage" error). When that goofy result occurred, I figured that I'd done something wrong, and then stumbled into the debug capability "built into" your configure scripting, so I tried that next, which ends up with the same result.
So, I don't know if this is yet another "goofy Solaris problem", but some aspect of enabling the debug symbols, seems to frak up some aspect of processing the command line at startup.
If you have any other ideas or thoughts, I'll certainly take them. I appreciate your time and input.
Anyway, I can use (for Solaris) libumem and mdb to narrow this down
to
a particular module, but without the debug symbols, I can’t really get to a specific chunk of code, so it would obviously help to get that working. If anyone has any ideas, I certainly would appreciate the input.
It'd really be appreciated if you could point out where the leak is happening. It might be use-case specific or some failure handler or whatever. Let's hope that starting syslog-ng from the command line works.
-- Bazsi
_______________________________________________________________________ _______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Balazs Scheidler
-
Balazs Scheidler
-
Marvin Nipper