[syslog-ng] syslog-ng memory usage grows

[Fabien Wernli]

Hi Jim,

On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote:
> I'm not sure what I am asking, other than general advice on:
> - performance using patterndb

performance is awesome, I wouldn't worry about it
although it would help to have some counters on the parsers in the stats
interface. That bieng said, it could help you to increase the stats level.

> - performance using redis destination

I can't comment on that one, sorry

> - advice on debugging where this memory growth is happening

Could you run syslog-ng inside `valgrind --tool=memcheck
--trace-children=yes --leak-check=yes`?

The problem is this will really slow down your instance, but let it run a
few minutes nevertheless and pastebin the result somewhere.

> As a rough measure - I have a syslog-ng process that has been running for
> less than 3 hours and right now is using 1.52GB of resident memory (shown by
> "top")

I'm suspecting `format-json` to be the source of the leak, or the flow
control cache. These have been the leak source of our own observations many
times, although many patches have solved the issues.

> (I would rather parse things in syslog-ng, but I *could* do this all using
> logstash/grok if this proves too much for patterndb at this load)

if you're worried about performance, I wouldn't even consider to begin thinking about comparing
LS/grok to patterndb