syslog-ng memory usage grows
Hi, I am continuing to test syslog-ng as the parser and shipper into an elasticsearch cluster. Right now I have syslog-ng 3.6.1 receiving logs at about ~7000 EPS and running them through a patterndb parser that splits out the (bluecoat proxy) fields into key-value pairs, writing them to a redis destination using format-json. syslog-ng 3.6.1 Installer-Version: 3.6.1 Revision: Compile-Date: Dec 9 2014 19:42:20 Available-Modules: dbparser,json-plugin,afuser,affile,afmongodb,tfgeoip,afprog,redis,afstomp,afsql,afsocket,system-source,afamqp,pseudofile,confgen,afsocket-tls,csvparser,basicfuncs,graphite,syslogformat,afsocket-notls,linux-kmsg-format,cryptofuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off I then use logstash to pull from redis and feed elasticsearch (the thought being that this would provide a buffer for the messages) Over the weekend I had syslog-ng crash (unfortunately no core) but now that I am watching it more closely, I see what appears to be continuous growth in memory usage (which leads me to suppose this was the cause of the crash). I'm not sure what I am asking, other than general advice on: - performance using patterndb - performance using redis destination - advice on debugging where this memory growth is happening As a rough measure - I have a syslog-ng process that has been running for less than 3 hours and right now is using 1.52GB of resident memory (shown by "top") I am including two config files. One has a single source, no filtering, single destination. The other splits logs from three upstream syslog-ng servers thinking this would provide "threading" with multiple patterndb parsers (I have read that patterndb is single threaded). Any thoughts on how to figure out where the memory is going? Or recommendations on whether or not using patterndb in this way is a particularly dumb idea :-) (I would rather parse things in syslog-ng, but I *could* do this all using logstash/grok if this proves too much for patterndb at this load) Thanks! Jim
Hi Jim, On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote:
I'm not sure what I am asking, other than general advice on: - performance using patterndb
performance is awesome, I wouldn't worry about it although it would help to have some counters on the parsers in the stats interface. That bieng said, it could help you to increase the stats level.
- performance using redis destination
I can't comment on that one, sorry
- advice on debugging where this memory growth is happening
Could you run syslog-ng inside `valgrind --tool=memcheck --trace-children=yes --leak-check=yes`? The problem is this will really slow down your instance, but let it run a few minutes nevertheless and pastebin the result somewhere.
As a rough measure - I have a syslog-ng process that has been running for less than 3 hours and right now is using 1.52GB of resident memory (shown by "top")
I'm suspecting `format-json` to be the source of the leak, or the flow control cache. These have been the leak source of our own observations many times, although many patches have solved the issues.
(I would rather parse things in syslog-ng, but I *could* do this all using logstash/grok if this proves too much for patterndb at this load)
if you're worried about performance, I wouldn't even consider to begin thinking about comparing LS/grok to patterndb
Hi Fabien, Hope the season is treating you well. I ran syslog-ng for 10 minutes (recall, ~7000 EPS incoming) under valgrind and here is the output (attached). It looks like there is some sort of leak as the process from yesterday had grown to over 16GB before I stopped it for the valgrind test. Thanks for the help (as always). And please let me know if I can provide any other data. Jim ---- Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Jim,
On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote:
I'm not sure what I am asking, other than general advice on: - performance using patterndb
performance is awesome, I wouldn't worry about it although it would help to have some counters on the parsers in the stats interface. That bieng said, it could help you to increase the stats level.
- performance using redis destination
I can't comment on that one, sorry
- advice on debugging where this memory growth is happening
Could you run syslog-ng inside `valgrind --tool=memcheck --trace-children=yes --leak-check=yes`?
The problem is this will really slow down your instance, but let it run a few minutes nevertheless and pastebin the result somewhere.
As a rough measure - I have a syslog-ng process that has been running for less than 3 hours and right now is using 1.52GB of resident memory (shown by "top")
I'm suspecting `format-json` to be the source of the leak, or the flow control cache. These have been the leak source of our own observations many times, although many patches have solved the issues.
(I would rather parse things in syslog-ng, but I *could* do this all using logstash/grok if this proves too much for patterndb at this load)
if you're worried about performance, I wouldn't even consider to begin thinking about comparing LS/grok to patterndb
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi All, I checked your valgrind log and the redis module and I think the leak is in redis.c:137: redisCommand(self->c, "ping"); This returns a reply object, but it's ignored and never freed. If we make a patch on GitHub, could you test it? Regards, Tibor 2014-12-17 18:36 GMT+01:00 <jrhendri@roadrunner.com>:
Hi Fabien,
Hope the season is treating you well.
I ran syslog-ng for 10 minutes (recall, ~7000 EPS incoming) under valgrind and here is the output (attached).
It looks like there is some sort of leak as the process from yesterday had grown to over 16GB before I stopped it for the valgrind test.
Thanks for the help (as always).
And please let me know if I can provide any other data.
Jim
---- Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Jim,
On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote:
I'm not sure what I am asking, other than general advice on: - performance using patterndb
performance is awesome, I wouldn't worry about it although it would help to have some counters on the parsers in the stats interface. That bieng said, it could help you to increase the stats level.
- performance using redis destination
I can't comment on that one, sorry
- advice on debugging where this memory growth is happening
Could you run syslog-ng inside `valgrind --tool=memcheck --trace-children=yes --leak-check=yes`?
The problem is this will really slow down your instance, but let it run a few minutes nevertheless and pastebin the result somewhere.
As a rough measure - I have a syslog-ng process that has been running for less than 3 hours and right now is using 1.52GB of resident memory (shown by "top")
I'm suspecting `format-json` to be the source of the leak, or the flow control cache. These have been the leak source of our own observations many times, although many patches have solved the issues.
(I would rather parse things in syslog-ng, but I *could* do this all using logstash/grok if this proves too much for patterndb at this load)
if you're worried about performance, I wouldn't even consider to begin thinking about comparing LS/grok to patterndb
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Fabien Wernli
-
Jim Hendrick
-
jrhendri＠roadrunner.com
-
Tibor Benke