[Bug 160] New: memory leak in message buffering
https://bugzilla.balabit.com/show_bug.cgi?id=160 Summary: memory leak in message buffering Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: erempel@uvic.ca Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=47) --> (https://bugzilla.balabit.com/attachment.cgi?id=47) tunable read rate program destination When writing to a destination that requires syslog-ng to buffer messages, the memory footprint of syslog-ng grows endlessly, even when there is sufficient time to flush the buffers. I am including a read rate tunable perl program destination that I use to slow the destination down. I am including a syslog-ng.conf file I am including the output of valgrind. I run syslog-ng, cat a /var/messages file (100,000 lines) to the input pipe. valgrind now shows definitely lost: 2,620,649 bytes in 4,439 blocks -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 --- Comment #1 from Evan Rempel <erempel@uvic.ca> 2012-02-13 19:16:20 --- Created an attachment (id=48) --> (https://bugzilla.balabit.com/attachment.cgi?id=48) syslog-ng.conf file to reproduce the problem. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 --- Comment #2 from Evan Rempel <erempel@uvic.ca> 2012-02-13 19:16:55 --- valgrind log can be found at http://web.uvic.ca/~erempel/valgrind-syslog-ng.20120213 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 Evan Rempel <erempel@uvic.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |critical -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Type of the Report|--- |regression CC| |algernon@balabit.hu AssignedTo|bazsi@balabit.hu |algernon@balabit.hu --- Comment #3 from Gergely Nagy <algernon@balabit.hu> 2012-02-14 10:25:40 --- Thanks for the valgrind log and the config, I'll have a look! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 Balazs Scheidler <bazsi@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bazsi@balabit.hu --- Comment #4 from Balazs Scheidler <bazsi@balabit.hu> 2012-02-28 20:46:51 --- This is odd, I could finally reproduce some kind of leak, but locally not all messages are leaked. I'd have to do some more testing to diagnose the issue, I just wanted to let you know that I was working on this. ==26035== 378,508 (328,236 direct, 50,272 indirect) bytes in 905 blocks are definitely lost in loss record 622 of 622 ==26035== at 0x4C28F9F: malloc (vg_replace_malloc.c:236) ==26035== by 0x591AF4F: g_malloc (gmem.c:131) ==26035== by 0x4E5BD46: log_msg_new (logmsg.c:979) ==26035== by 0x4E62B05: log_reader_work_perform (logreader.c:493) ==26035== by 0x4E63298: log_reader_io_process_input (logreader.c:191) ==26035== by 0x4E87705: iv_run_tasks (iv_task.c:62) ==26035== by 0x4E8634C: iv_main (iv_main.c:253) ==26035== by 0x4E696CD: main_loop_run (mainloop.c:731) ==26035== by 0x4017EB: main (main.c:260) ==26035== The odd thing is that I've fed about 200000 messages and about 1000 was leaked. And I've stopped syslog-ng _after_ the full queue was sent to the perl program, so the queue should've been empty. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 Balazs Scheidler <bazsi@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|ASSIGNED |RESOLVED --- Comment #5 from Balazs Scheidler <bazsi@balabit.hu> 2012-03-11 14:54:57 --- This patch should fix this: commit 3c14a264ab4c76b9372b1288d6808d19ec4d0aed Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Mar 11 14:53:55 2012 +0100 logwriter: fixed a memory leak for messages that couldn't be flushed at once Sometimes a message is dequeued, and then can't be posted to the destination (probably because of an EAGAIN or something like that). In this case it is readded back to the queue. In this case a reference was lost, causing a memory leak, but not for all messages, but only for the affected ones. Reported-By: Evan Rempel <erempel@uvic.ca> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=160 --- Comment #6 from Evan Rempel <erempel@uvic.ca> 2012-03-12 20:31:37 --- I just wanted to confirm that this fixes the memory leak that we observed in our environment. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla＠bugzilla.balabit.com