[syslog-ng] syslog-ng memory usage grows

[Jim Hendrick]

I will certainly try.

Send me a link, and Thanks!



Sent from my Verizon Wireless 4G LTE smartphone


-------- Original message --------
From: Tibor Benke <ihrwein at gmail.com> 
Date:12/17/2014  5:42 PM  (GMT-05:00) 
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu> 
Cc: wernli at in2p3.fr, jrhendri at roadrunner.com 
Subject: Re: [syslog-ng] syslog-ng memory usage grows 

Hi All,

I checked your valgrind log and the redis module and I think the leak is in redis.c:137:

      redisCommand(self->c, "ping"); 

This returns a reply object, but it's ignored and never freed. 

If we make a patch on GitHub, could you test it?

Regards,
Tibor

2014-12-17 18:36 GMT+01:00 <jrhendri at roadrunner.com>:
Hi Fabien,

  Hope the season is treating you well.

  I ran syslog-ng for 10 minutes (recall, ~7000 EPS incoming) under valgrind and here is the output (attached).

  It looks like there is some sort of leak as the process from yesterday had grown to over 16GB before I stopped it for the valgrind test.

Thanks for the help (as always).

And please let me know if I can provide any other data.

Jim



---- Fabien Wernli <wernli at in2p3.fr> wrote:
> Hi Jim,
>
> On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote:
> > I'm not sure what I am asking, other than general advice on:
> > - performance using patterndb
>
> performance is awesome, I wouldn't worry about it
> although it would help to have some counters on the parsers in the stats
> interface. That bieng said, it could help you to increase the stats level.
>
> > - performance using redis destination
>
> I can't comment on that one, sorry
>
> > - advice on debugging where this memory growth is happening
>
> Could you run syslog-ng inside `valgrind --tool=memcheck
> --trace-children=yes --leak-check=yes`?
>
> The problem is this will really slow down your instance, but let it run a
> few minutes nevertheless and pastebin the result somewhere.
>
> > As a rough measure - I have a syslog-ng process that has been running for
> > less than 3 hours and right now is using 1.52GB of resident memory (shown by
> > "top")
>
> I'm suspecting `format-json` to be the source of the leak, or the flow
> control cache. These have been the leak source of our own observations many
> times, although many patches have solved the issues.
>
> > (I would rather parse things in syslog-ng, but I *could* do this all using
> > logstash/grok if this proves too much for patterndb at this load)
>
> if you're worried about performance, I wouldn't even consider to begin thinking about comparing
> LS/grok to patterndb
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141217/a35f5ff3/attachment.htm