[syslog-ng] syslog-ng memory usage grows

[Tibor Benke]

The patch seems to work, I sent a PR.

https://github.com/balabit/syslog-ng/pull/359

2014-12-18 8:48 GMT+01:00 Tibor Benke <ihrwein at gmail.com>:
>
> You can find the patch on this branch:
>
> https://github.com/ihrwein/syslog-ng/tree/f/fix-redis-memleak
>
>
>
> 2014-12-18 3:11 GMT+01:00 Jim Hendrick <jrhendri at roadrunner.com>:
>>
>> I will certainly try.
>>
>> Send me a link, and Thanks!
>>
>>
>>
>> Sent from my Verizon Wireless 4G LTE smartphone
>>
>>
>> -------- Original message --------
>> From: Tibor Benke <ihrwein at gmail.com>
>> Date:12/17/2014 5:42 PM (GMT-05:00)
>> To: Syslog-ng users' and developers' mailing list <
>> syslog-ng at lists.balabit.hu>
>> Cc: wernli at in2p3.fr, jrhendri at roadrunner.com
>> Subject: Re: [syslog-ng] syslog-ng memory usage grows
>>
>> Hi All,
>>
>> I checked your valgrind log and the redis module and I think the leak is
>> in redis.c:137:
>>
>>       redisCommand(self->c, "ping");
>>
>> This returns a reply object, but it's ignored and never freed.
>>
>> If we make a patch on GitHub, could you test it?
>>
>> Regards,
>> Tibor
>>
>> 2014-12-17 18:36 GMT+01:00 <jrhendri at roadrunner.com>:
>>>
>>> Hi Fabien,
>>>
>>>   Hope the season is treating you well.
>>>
>>>   I ran syslog-ng for 10 minutes (recall, ~7000 EPS incoming) under
>>> valgrind and here is the output (attached).
>>>
>>>   It looks like there is some sort of leak as the process from yesterday
>>> had grown to over 16GB before I stopped it for the valgrind test.
>>>
>>> Thanks for the help (as always).
>>>
>>> And please let me know if I can provide any other data.
>>>
>>> Jim
>>>
>>>
>>>
>>> ---- Fabien Wernli <wernli at in2p3.fr> wrote:
>>> > Hi Jim,
>>> >
>>> > On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote:
>>> > > I'm not sure what I am asking, other than general advice on:
>>> > > - performance using patterndb
>>> >
>>> > performance is awesome, I wouldn't worry about it
>>> > although it would help to have some counters on the parsers in the
>>> stats
>>> > interface. That bieng said, it could help you to increase the stats
>>> level.
>>> >
>>> > > - performance using redis destination
>>> >
>>> > I can't comment on that one, sorry
>>> >
>>> > > - advice on debugging where this memory growth is happening
>>> >
>>> > Could you run syslog-ng inside `valgrind --tool=memcheck
>>> > --trace-children=yes --leak-check=yes`?
>>> >
>>> > The problem is this will really slow down your instance, but let it
>>> run a
>>> > few minutes nevertheless and pastebin the result somewhere.
>>> >
>>> > > As a rough measure - I have a syslog-ng process that has been
>>> running for
>>> > > less than 3 hours and right now is using 1.52GB of resident memory
>>> (shown by
>>> > > "top")
>>> >
>>> > I'm suspecting `format-json` to be the source of the leak, or the flow
>>> > control cache. These have been the leak source of our own observations
>>> many
>>> > times, although many patches have solved the issues.
>>> >
>>> > > (I would rather parse things in syslog-ng, but I *could* do this all
>>> using
>>> > > logstash/grok if this proves too much for patterndb at this load)
>>> >
>>> > if you're worried about performance, I wouldn't even consider to begin
>>> thinking about comparing
>>> > LS/grok to patterndb
>>> >
>>> >
>>> ______________________________________________________________________________
>>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> > Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>> >
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141218/ec16db66/attachment.htm