[syslog-ng] 3.3.4 anyone else notice a memory leak

Sun Feb 12 04:10:41 CET 2012

OK, I think I can state that there is a memory problem. I would not classify it as a leak in that
valgrind does not show it, but that just means that the original pointer is not lost. It appears that
syslog-ng keeps track of all the memory, but it just keeps acquiring more.

I complied everything up with the Thread-Caching Malloc

http://code.google.com/p/gperftools/

which does its own malloc, and then manages it all within the user space of the application making it
much faster.

By doing this I remove the question of the OS doing lazy free and the mallocs just piling up waiting for
the OS to free them.

The memory footprint grows continuously.

The other thing that ctmalloc does is provide memory profiling tools. It might me interesting
for the syslog-ng developers to use this feature to assist in debugging memory problems like the one I
think I have.

I just made a pipe, made syslog-ng read from it and write to a program destination of 'cat > /dev/null'

Then cat a large file (200,000 lines, I used an exisitng syslog file) to the pipe. Watch the memory grow.
Repeat the process and watch the memory grow more.
Seems to grow by about 3 MB every time I cat the file.

Evan.
________________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Evan Rempel [erempel at uvic.ca]
Sent: Saturday, February 11, 2012 11:32 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak

No, that does not help the issue at all.

It almost seems to be that the OS is doing a lazy malloc/free and that syslog-ng is free(ing) the
memory, but the OS keeps is associated with the application. Then, if syslog-ng needs to buffer some
messages again it does a new malloc and the OS gives it a different block of with the end result that
the memory footprint continues to grow.

I saw this in 3.0 where the RAM footprint would grow when syslog-ng buffered memory, but
it would take 3-4 hours after the destination was available and the buffer completely flushed before
the RAM footprint would start to shrink and another 3-4 hours before it reached the original size
prior to the buffering.

I'm going to play with the caching malloc to see if that makes a difference, but with syslog-ng allocating memory
blocks of differing sizes I'm not confident that the caching malloc will be able to do anything.

Evan.

________________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Jakub Jankowski [shasta at toxcorp.com]
Sent: Friday, February 10, 2012 4:10 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak

On 2012-02-10, Evan Rempel wrote:

>>> I am fairly certain that there is a memory leak. How do I help track
>>> it down?
[...]
> Yup, I can reproduce it. The output of valgrind does not appear to
> show the leak. If I run syslog-ng and do NOT send any data to it (it
> processes no
> log lines) the output of valgrind is the same as if it processes 100,000 log
> lines,
> however, when I "dump" 100,000 log lines
>
> % wc /home1l/erempel/log
>  107590  1233928 18956930 /home1l/erempel/log
>
> into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of
> syslog-ng
> grows by 3MB, every time I dump in the log lines.

Okay, this might be a long shot, but... can you try to revert commit [1],
recompile syslog-ng and reproduce the leak? (grab the tarball, unpack it,
get the raw form of [1], and use -R option to reverse the patch; then
your usual build procedure)

This commit was applied right after 3.3.2 was released, because it
supposedly fixed crash with tcp destinations. Bazsi had doubts[2] that it
may reintroduce a memory leak... and now I know it *does* reintroduce the
leak, because it was leak I originally reported (longish thread here, see
[3], [4], [5]) and now I'm seeing it again, with both 3.3.3 and 3.3.4:

==3645== LEAK SUMMARY:
==3645==    definitely lost: 3,719,421 bytes in 10,008 blocks
==3645==    indirectly lost: 905,917 bytes in 10,014 blocks
==3645==      possibly lost: 17,214 bytes in 128 blocks
==3645==    still reachable: 80,063 bytes in 3,278 blocks
==3645==         suppressed: 0 bytes in 0 blocks

I've just re-ran my tests[6] with commit [1] reverted, and the results are
good again:

==7885== LEAK SUMMARY:
==7885==    definitely lost: 297 bytes in 6 blocks
==7885==    indirectly lost: 1,025 bytes in 22 blocks
==7885==      possibly lost: 13,527 bytes in 110 blocks
==7885==    still reachable: 106,235 bytes in 3,327 blocks
==7885==         suppressed: 0 bytes in 0 blocks

I understand that my use-case scenario is a bit different than yours, but
perhaps the underlying issue is the same?

Anyway - Bazsi, Gergely, could you please take a look at this whole thing?
Because looks like we either end up with huge memleaks (with [1]) or
tcp()-related crashes (without it).

Hope this helps a bit.

[1] http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commitdiff;h=c7070e2a6f1c3a312260bcecf49d62028fef27ce
[2] https://lists.balabit.hu/pipermail/syslog-ng/2011-November/017697.html
[3] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017269.html
[4] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017275.html
[5] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017338.html
[6] https://lists.balabit.hu/pipermail/syslog-ng/2011-October/017555.html

Regards,
  Jakub

--
Jakub Jankowski|shasta at toxcorp.com|http://toxcorp.com/
GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq