syslog-ng memory leak ? valgrind attached
From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim
I had the same issue, but couldn't reproduce it when running under valgrind. I wasn't using json and redis, either. I submitted https://github.com/balabit/syslog-ng/issues/446 <https://github.com/balabit/syslog-ng/issues/446> for this, which includes a copy of my config file. I need to set up a test environment so that reproducing the issue doesn't require me to lose real messages before I can proceed further, but maybe you can help out. -Pradeep
On Apr 29, 2015, at 7:19 PM, Jim Hendrick <jrhendri@roadrunner.com> wrote:
From 3.6.2 OSE built with json & redis
Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load)
This was from a few minutes of runtime. (~5)
Is this a known issue or did I trip over something?
Thanks! Jim
<valgrind.err.gz>______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Jim, On Wed, Apr 29, 2015 at 09:49:05AM -0400, Jim Hendrick wrote:
From 3.6.2 OSE built with json & redis
Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load)
This was from a few minutes of runtime. (~5)
Is this a known issue or did I trip over something?
We're having similar issues when the workload tickles your figures. One possible explanation seems to be that the memory from queues is never released [1]. Are your queues always empty or do they do regular work? Oh, then there was this one too [2] [1] https://github.com/balabit/syslog-ng/issues/287 [2] https://github.com/balabit/syslog-ng/issues/446
For the redis destination it's a know problem, was quickly fixed after 3.6.2 release: https://github.com/balabit/syslog-ng/commit/53606941b1b6b6726c4945f3560984a7... If you use my unofficial syslog-ng packages, let me know, and I can add this patch. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Wed, Apr 29, 2015 at 3:49 PM, Jim Hendrick <jrhendri@roadrunner.com> wrote:
From 3.6.2 OSE built with json & redis
Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load)
This was from a few minutes of runtime. (~5)
Is this a known issue or did I trip over something?
Thanks! Jim
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Also LD_PRELOAD jemalloc could improve the situation a lot. Ive seen cases where our usage patterns created a lot of memory use (probably because of fragmentation but I could not check), and just using jemalloc solved the problem. On Apr 29, 2015 3:49 PM, "Jim Hendrick" <jrhendri@roadrunner.com> wrote:
From 3.6.2 OSE built with json & redis
Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load)
This was from a few minutes of runtime. (~5)
Is this a known issue or did I trip over something?
Thanks! Jim
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Czanik, Péter
-
Fabien Wernli
-
Jim Hendrick
-
Pradeep Sanders
-
Scheidler, Balázs