No, that does not help the issue at all. It almost seems to be that the OS is doing a lazy malloc/free and that syslog-ng is free(ing) the memory, but the OS keeps is associated with the application. Then, if syslog-ng needs to buffer some messages again it does a new malloc and the OS gives it a different block of with the end result that the memory footprint continues to grow. I saw this in 3.0 where the RAM footprint would grow when syslog-ng buffered memory, but it would take 3-4 hours after the destination was available and the buffer completely flushed before the RAM footprint would start to shrink and another 3-4 hours before it reached the original size prior to the buffering. I'm going to play with the caching malloc to see if that makes a difference, but with syslog-ng allocating memory blocks of differing sizes I'm not confident that the caching malloc will be able to do anything. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jakub Jankowski [shasta@toxcorp.com] Sent: Friday, February 10, 2012 4:10 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak On 2012-02-10, Evan Rempel wrote:
I am fairly certain that there is a memory leak. How do I help track it down? [...] Yup, I can reproduce it. The output of valgrind does not appear to show the leak. If I run syslog-ng and do NOT send any data to it (it processes no log lines) the output of valgrind is the same as if it processes 100,000 log lines, however, when I "dump" 100,000 log lines
% wc /home1l/erempel/log 107590 1233928 18956930 /home1l/erempel/log
into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of syslog-ng grows by 3MB, every time I dump in the log lines.
Okay, this might be a long shot, but... can you try to revert commit [1], recompile syslog-ng and reproduce the leak? (grab the tarball, unpack it, get the raw form of [1], and use -R option to reverse the patch; then your usual build procedure) This commit was applied right after 3.3.2 was released, because it supposedly fixed crash with tcp destinations. Bazsi had doubts[2] that it may reintroduce a memory leak... and now I know it *does* reintroduce the leak, because it was leak I originally reported (longish thread here, see [3], [4], [5]) and now I'm seeing it again, with both 3.3.3 and 3.3.4: ==3645== LEAK SUMMARY: ==3645== definitely lost: 3,719,421 bytes in 10,008 blocks ==3645== indirectly lost: 905,917 bytes in 10,014 blocks ==3645== possibly lost: 17,214 bytes in 128 blocks ==3645== still reachable: 80,063 bytes in 3,278 blocks ==3645== suppressed: 0 bytes in 0 blocks I've just re-ran my tests[6] with commit [1] reverted, and the results are good again: ==7885== LEAK SUMMARY: ==7885== definitely lost: 297 bytes in 6 blocks ==7885== indirectly lost: 1,025 bytes in 22 blocks ==7885== possibly lost: 13,527 bytes in 110 blocks ==7885== still reachable: 106,235 bytes in 3,327 blocks ==7885== suppressed: 0 bytes in 0 blocks I understand that my use-case scenario is a bit different than yours, but perhaps the underlying issue is the same? Anyway - Bazsi, Gergely, could you please take a look at this whole thing? Because looks like we either end up with huge memleaks (with [1]) or tcp()-related crashes (without it). Hope this helps a bit. [1] http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commitdiff;h=c7070e2a6f1c... [2] https://lists.balabit.hu/pipermail/syslog-ng/2011-November/017697.html [3] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017269.html [4] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017275.html [5] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017338.html [6] https://lists.balabit.hu/pipermail/syslog-ng/2011-October/017555.html Regards, Jakub -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq