Hi, I am continuing to test syslog-ng as the parser and shipper into an elasticsearch cluster. Right now I have syslog-ng 3.6.1 receiving logs at about ~7000 EPS and running them through a patterndb parser that splits out the (bluecoat proxy) fields into key-value pairs, writing them to a redis destination using format-json. syslog-ng 3.6.1 Installer-Version: 3.6.1 Revision: Compile-Date: Dec 9 2014 19:42:20 Available-Modules: dbparser,json-plugin,afuser,affile,afmongodb,tfgeoip,afprog,redis,afstomp,afsql,afsocket,system-source,afamqp,pseudofile,confgen,afsocket-tls,csvparser,basicfuncs,graphite,syslogformat,afsocket-notls,linux-kmsg-format,cryptofuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off I then use logstash to pull from redis and feed elasticsearch (the thought being that this would provide a buffer for the messages) Over the weekend I had syslog-ng crash (unfortunately no core) but now that I am watching it more closely, I see what appears to be continuous growth in memory usage (which leads me to suppose this was the cause of the crash). I'm not sure what I am asking, other than general advice on: - performance using patterndb - performance using redis destination - advice on debugging where this memory growth is happening As a rough measure - I have a syslog-ng process that has been running for less than 3 hours and right now is using 1.52GB of resident memory (shown by "top") I am including two config files. One has a single source, no filtering, single destination. The other splits logs from three upstream syslog-ng servers thinking this would provide "threading" with multiple patterndb parsers (I have read that patterndb is single threaded). Any thoughts on how to figure out where the memory is going? Or recommendations on whether or not using patterndb in this way is a particularly dumb idea :-) (I would rather parse things in syslog-ng, but I *could* do this all using logstash/grok if this proves too much for patterndb at this load) Thanks! Jim