[syslog-ng] 3.3.4 anyone else notice a memory leak

[Evan Rempel]

No, that does not help the issue at all.

It almost seems to be that the OS is doing a lazy malloc/free and that syslog-ng is free(ing) the
memory, but the OS keeps is associated with the application. Then, if syslog-ng needs to buffer some
messages again it does a new malloc and the OS gives it a different block of with the end result that
the memory footprint continues to grow.

I saw this in 3.0 where the RAM footprint would grow when syslog-ng buffered memory, but
it would take 3-4 hours after the destination was available and the buffer completely flushed before
the RAM footprint would start to shrink and another 3-4 hours before it reached the original size
prior to the buffering.

I'm going to play with the caching malloc to see if that makes a difference, but with syslog-ng allocating memory
blocks of differing sizes I'm not confident that the caching malloc will be able to do anything.

Evan.

________________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Jakub Jankowski [shasta at toxcorp.com]
Sent: Friday, February 10, 2012 4:10 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak

On 2012-02-10, Evan Rempel wrote:

>>> I am fairly certain that there is a memory leak. How do I help track
>>> it down?
[...]
> Yup, I can reproduce it. The output of valgrind does not appear to
> show the leak. If I run syslog-ng and do NOT send any data to it (it
> processes no
> log lines) the output of valgrind is the same as if it processes 100,000 log
> lines,
> however, when I "dump" 100,000 log lines
>
> % wc /home1l/erempel/log
>  107590  1233928 18956930 /home1l/erempel/log
>
> into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of
> syslog-ng
> grows by 3MB, every time I dump in the log lines.

Okay, this might be a long shot, but... can you try to revert commit [1],
recompile syslog-ng and reproduce the leak? (grab the tarball, unpack it,
get the raw form of [1], and use -R option to reverse the patch; then
your usual build procedure)

This commit was applied right after 3.3.2 was released, because it
supposedly fixed crash with tcp destinations. Bazsi had doubts[2] that it
may reintroduce a memory leak... and now I know it *does* reintroduce the
leak, because it was leak I originally reported (longish thread here, see
[3], [4], [5]) and now I'm seeing it again, with both 3.3.3 and 3.3.4:

==3645== LEAK SUMMARY:
==3645==    definitely lost: 3,719,421 bytes in 10,008 blocks
==3645==    indirectly lost: 905,917 bytes in 10,014 blocks
==3645==      possibly lost: 17,214 bytes in 128 blocks
==3645==    still reachable: 80,063 bytes in 3,278 blocks
==3645==         suppressed: 0 bytes in 0 blocks


I've just re-ran my tests[6] with commit [1] reverted, and the results are
good again:

==7885== LEAK SUMMARY:
==7885==    definitely lost: 297 bytes in 6 blocks
==7885==    indirectly lost: 1,025 bytes in 22 blocks
==7885==      possibly lost: 13,527 bytes in 110 blocks
==7885==    still reachable: 106,235 bytes in 3,327 blocks
==7885==         suppressed: 0 bytes in 0 blocks


I understand that my use-case scenario is a bit different than yours, but
perhaps the underlying issue is the same?

Anyway - Bazsi, Gergely, could you please take a look at this whole thing?
Because looks like we either end up with huge memleaks (with [1]) or
tcp()-related crashes (without it).

Hope this helps a bit.

[1] http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commitdiff;h=c7070e2a6f1c3a312260bcecf49d62028fef27ce
[2] https://lists.balabit.hu/pipermail/syslog-ng/2011-November/017697.html
[3] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017269.html
[4] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017275.html
[5] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017338.html
[6] https://lists.balabit.hu/pipermail/syslog-ng/2011-October/017555.html


Regards,
  Jakub

--
Jakub Jankowski|shasta at toxcorp.com|http://toxcorp.com/
GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq