Hi, Thanks for the detailed problem description Jakub and the diagnosis Gergely. I've committed this patch to master: commit 2ed7f92153aec5e4e666ea17dcd8d2232a8e76f9 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Tue Sep 27 20:12:24 2011 +0200 log_dest_driver_release_queue(): unref the queue even if no name is present This patch fixes a memory leak for file destinations, in which we leak all LogQueue instances, possibly containing data, which can become quite large, especially if a lot of log files are opened/closed continously. Reported-By: Jakub Jankowski <shasta@toxcorp.com> Cc: <syslog-ng-stable@balabit.hu> Cc: Gergely Nagy <algernon@balabit.hu> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> While I was there, I've also fixed a related problem, although less likely to happen: commit e224d45da2ecad68f7f9c44895849a0fc795aae5 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Tue Sep 27 20:14:35 2011 +0200 log_dest_driver_acquire_queue(): make sure all dest drivers have a name The persist_name for acquire_queue() identifies the destination driver that would like to keep its queue kept accross reloads. File destinations didn't keep their queue, effectively causing file drivers to lose their queue accross reloads. This went unnoticed as LogWriter does one last time to flush everything out to the disk when reloading, but still it is more important to keep the queue properly. Cc: syslog-ng-stable@balabit.hu Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> On Mon, 2011-09-19 at 14:16 +0200, Gergely Nagy wrote:
Jakub Jankowski <shasta@toxcorp.com> writes:
It looks like 3.2.2 does not exhibit the leak I'm seeing on 3.3. Complete valgrind logs are available here: http://toxcorp.com/stuff/syslog-ng-leak/
Let me know if I can provide anything else to help with this. Thanks!
Quick update: I managed to reproduce the problem, and I can verify that this is present in current git head indeed.
I used the following config:
,---- | @version: 3.3 | @include "scl.conf" | | options { | time-reap(1); | create-dirs(yes); | frac-digits(4); | }; | source s_network { | tcp(port (13514) tags("tcp-tag") flags(no-parse)); | }; | destination d_file { | file("/tmp/many-file/${FIRST:-default}-seq-${UNIXTIME}.log" | template("${REST}\n") | ); | }; | log { | source(s_network); | destination(d_file); | }; `----
Then do the following: for f in $(seq 1 10000); do echo "host-$f message $f" | nc localhost 13514; done
I have an idea where to look further, will see how far I get.
-- Bazsi