Heavy memory leak using correlation or patterndb
Hello. Since I installed the rules dedicated to dropping LDAP probe connection traces, the memory usage of my ldap server exploded. top confirms than syslog-ng is the culprit, using 60% of the available memory. Here is a weekly munin graph presenting the issue: http://www.zarb.org/~guillomovitch/memory-week.png The rules have been installed on 19th, and I restarted syslog-ng yesterday. I'm using syslog-ng 3.2.1, on mandriva 2010.0, 64 bits. Here are the exact rules used, if that matters (I couldn't try latest Balazs suggestions yet). The context-timeout for the first rule is quite low, because the probe checks every two seconds. By any chance, doesn't the second timeout, set to 0, set an illimited session expiration time ? If not the case, I'd gladly try any needed experiment, such as running syslog-ng under valgrind, to identify the leak source. -- BOFH excuse #400: We are Microsoft. What you are experiencing is not a problem; it is an undocumented feature.
Le 26/01/2011 11:01, Guillaume Rousse a écrit :
By any chance, doesn't the second timeout, set to 0, set an illimited session expiration time ? I just tried with context-timeout set to 1, memory usage stays constant, so it seems my hypothesis was correct. -- BOFH excuse #303:
fractal radiation jamming the backbone
On Wed, 2011-01-26 at 11:01 +0100, Guillaume Rousse wrote:
Hello.
Since I installed the rules dedicated to dropping LDAP probe connection traces, the memory usage of my ldap server exploded. top confirms than syslog-ng is the culprit, using 60% of the available memory. Here is a weekly munin graph presenting the issue: http://www.zarb.org/~guillomovitch/memory-week.png
The rules have been installed on 19th, and I restarted syslog-ng yesterday.
I'm using syslog-ng 3.2.1, on mandriva 2010.0, 64 bits. Here are the exact rules used, if that matters (I couldn't try latest Balazs suggestions yet). The context-timeout for the first rule is quite low, because the probe checks every two seconds. By any chance, doesn't the second timeout, set to 0, set an illimited session expiration time ?
If not the case, I'd gladly try any needed experiment, such as running syslog-ng under valgrind, to identify the leak source.
Hmm.. thanks for the report, and sorry for not going after it any sooner. I'll try to reproduce the problem as no immediate culprit is visible without reproduction. I hope to get it fixed right after I send this email, so that hopefully means today. If not, then I'm not sure when I'll have a chance to work on it again. :( -- Bazsi
On Sat, 2011-02-05 at 15:47 +0100, Balazs Scheidler wrote:
On Wed, 2011-01-26 at 11:01 +0100, Guillaume Rousse wrote:
Hello.
Since I installed the rules dedicated to dropping LDAP probe connection traces, the memory usage of my ldap server exploded. top confirms than syslog-ng is the culprit, using 60% of the available memory. Here is a weekly munin graph presenting the issue: http://www.zarb.org/~guillomovitch/memory-week.png
The rules have been installed on 19th, and I restarted syslog-ng yesterday.
I'm using syslog-ng 3.2.1, on mandriva 2010.0, 64 bits. Here are the exact rules used, if that matters (I couldn't try latest Balazs suggestions yet). The context-timeout for the first rule is quite low, because the probe checks every two seconds. By any chance, doesn't the second timeout, set to 0, set an illimited session expiration time ?
If not the case, I'd gladly try any needed experiment, such as running syslog-ng under valgrind, to identify the leak source.
Hmm.. thanks for the report, and sorry for not going after it any sooner. I'll try to reproduce the problem as no immediate culprit is visible without reproduction. I hope to get it fixed right after I send this email, so that hopefully means today.
If not, then I'm not sure when I'll have a chance to work on it again. :(
I wasn't able to reproduce it either with current git HEAD (3.2.2 + unrelated fixes) or v3.2.1. I was using your example messages to produce a file with 20000 log messages and then told pdbtool to read the file using pdbtool match, and then even fed the messages to syslog-ng using a db-parser() with your patterndb. Neither cases did it start to grow. I've checked using valgrind, but nothing suspicios. So your setup is definitely affecting it somehow. Can you please try to reproduce it with pdbtool match first (that would be better and easier to diagnose) with a concrete logfile that is processed by your patterndb rules? If that doesn't start growing, then the only chance is to catch syslog-ng while it is growing using valgrind. A preferred command line to do that would be: G_SLICE=always-malloc valgrind --leak-check=yes --show-reachable=yes /sbin/syslog-ng ... Thanks for your efforts for helping me to track this down. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Guillaume Rousse