[syslog-ng] syslog-ng memory usage grows

[Tibor Benke]

You can find the patch on this branch:

https://github.com/ihrwein/syslog-ng/tree/f/fix-redis-memleak



2014-12-18 3:11 GMT+01:00 Jim Hendrick <jrhendri at roadrunner.com>:
>
> I will certainly try.
>
> Send me a link, and Thanks!
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: Tibor Benke <ihrwein at gmail.com>
> Date:12/17/2014 5:42 PM (GMT-05:00)
> To: Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> Cc: wernli at in2p3.fr, jrhendri at roadrunner.com
> Subject: Re: [syslog-ng] syslog-ng memory usage grows
>
> Hi All,
>
> I checked your valgrind log and the redis module and I think the leak is
> in redis.c:137:
>
>       redisCommand(self->c, "ping");
>
> This returns a reply object, but it's ignored and never freed.
>
> If we make a patch on GitHub, could you test it?
>
> Regards,
> Tibor
>
> 2014-12-17 18:36 GMT+01:00 <jrhendri at roadrunner.com>:
>>
>> Hi Fabien,
>>
>>   Hope the season is treating you well.
>>
>>   I ran syslog-ng for 10 minutes (recall, ~7000 EPS incoming) under
>> valgrind and here is the output (attached).
>>
>>   It looks like there is some sort of leak as the process from yesterday
>> had grown to over 16GB before I stopped it for the valgrind test.
>>
>> Thanks for the help (as always).
>>
>> And please let me know if I can provide any other data.
>>
>> Jim
>>
>>
>>
>> ---- Fabien Wernli <wernli at in2p3.fr> wrote:
>> > Hi Jim,
>> >
>> > On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote:
>> > > I'm not sure what I am asking, other than general advice on:
>> > > - performance using patterndb
>> >
>> > performance is awesome, I wouldn't worry about it
>> > although it would help to have some counters on the parsers in the stats
>> > interface. That bieng said, it could help you to increase the stats
>> level.
>> >
>> > > - performance using redis destination
>> >
>> > I can't comment on that one, sorry
>> >
>> > > - advice on debugging where this memory growth is happening
>> >
>> > Could you run syslog-ng inside `valgrind --tool=memcheck
>> > --trace-children=yes --leak-check=yes`?
>> >
>> > The problem is this will really slow down your instance, but let it run
>> a
>> > few minutes nevertheless and pastebin the result somewhere.
>> >
>> > > As a rough measure - I have a syslog-ng process that has been running
>> for
>> > > less than 3 hours and right now is using 1.52GB of resident memory
>> (shown by
>> > > "top")
>> >
>> > I'm suspecting `format-json` to be the source of the leak, or the flow
>> > control cache. These have been the leak source of our own observations
>> many
>> > times, although many patches have solved the issues.
>> >
>> > > (I would rather parse things in syslog-ng, but I *could* do this all
>> using
>> > > logstash/grok if this proves too much for patterndb at this load)
>> >
>> > if you're worried about performance, I wouldn't even consider to begin
>> thinking about comparing
>> > LS/grok to patterndb
>> >
>> >
>> ______________________________________________________________________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141218/5b155c95/attachment.htm