On Sun, 2011-10-09 at 23:16 +0200, Jakub Jankowski wrote:
On 2011-10-09, Jakub Jankowski wrote:
On 2011-10-09, Balazs Scheidler wrote:
On Wed, 2011-10-05 at 01:44 +0200, Jakub Jankowski wrote:
On Mon, 3 Oct 2011 14:34:13 +0200 (CEST), Jakub Jankowski wrote:
I think I spoke too soon. :-/ See attached cacti graph. This is basically an output of awk '/VmRSS:/ { $print $2*1024 }' /proc/$(cat /var/run/syslog-ng.pid)/status on git head ("3.3.1" ATM). There must be something else going on; any hints on how to diagnose this? It definitely seems to be a leak of some kind. If at all possible, running it for some time under valgrind like you did previously would be tremendous. I've run half my "live" traffic for approx. 5 minutes through another node, with syslog-ng 3.3.1 running under valgrind. Log available at: http://toxcorp.com/stuff/syslog-ng-leak/s3.3.1-prod-half-5min.log Does it ring any bells (it doesn't for me). Should I test it longer?
I've let it run ~83 minutes this time. No substantial changes in valgrind output: http://toxcorp.com/stuff/syslog-ng-leak/s3.3.1-prod-half-83min.log What baffles me is the number of errors related to "Syscall param epoll_ctl(event) points to uninitialised byte(s)".
This is not nice, but shouldn't cause harm, but I'll recheck.
Anything else looks interesting?
During this run, VmRSS kept rising, from: Sun Oct 9 21:44:51 CEST 2011; VmRSS: 3804 kB Sun Oct 9 21:44:52 CEST 2011; VmRSS: 3816 kB Sun Oct 9 21:44:53 CEST 2011; VmRSS: 3820 kB Sun Oct 9 21:44:54 CEST 2011; VmRSS: 3844 kB (...) to: Sun Oct 9 23:07:37 CEST 2011; VmRSS: 444760 kB Sun Oct 9 23:07:38 CEST 2011; VmRSS: 444764 kB Sun Oct 9 23:07:39 CEST 2011; VmRSS: 444764 kB Sun Oct 9 23:07:40 CEST 2011; VmRSS: 444880 kB Sun Oct 9 23:07:41 CEST 2011; VmRSS: 444896 kB Sun Oct 9 23:07:42 CEST 2011; VmRSS: 444896 kB Sun Oct 9 23:07:43 CEST 2011; VmRSS: 444896 kB Sun Oct 9 23:07:44 CEST 2011; VmRSS: 445332 kB
Valgrind itself allocates memory to store the errors and warnings. Do you see this increase of memory usage without valgrind too? The valgrind output doesn't look bad, 97k leaked in 83 minutes, most of which seem to be one-off allocation which would be freed if syslog-ng was terminated properly. in use at exit: 97,299 bytes in 3,425 blocks I've rechecked the epoll_ctl() error, and as it seems vagrind doesn't like that some of the bytes in the epoll_event structure aren't initialized. However they don't need to be initialized, and I wouldn't want to add a memset() call. Hopefully valgrind will be fixed. It is possible to suppress these somehow. I've also read the complete log and I've found two minor leaks, both of which leak memory when processing SIGHUPs, I've fixed these: commit 0136e1b8a0460dd34be74c3fe6e202f971484be4 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Thu Oct 13 15:28:57 2011 +0200 afsql: fixed a memory leak of the "indexes" array This bug caused some memory leaked at every SIGHUP. Reported-By: Jakub Jankowski <shasta@toxcorp.com> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> commit 5ef8dbba059d4670896e81f0bcf2bf63686f119b Author: Balazs Scheidler <bazsi@balabit.hu> Date: Thu Oct 13 15:28:17 2011 +0200 afuser: fixed the leakage of the username The afuser driver was leaking the name of the user, accumulating leaked some memory at every SIGHUP. Reported-By: Jakub Jankowski <shasta@toxcorp.com> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> The rest seems to be syslog-ng operating properly. Does it exhibit the same behaviour w/o valgrind? -- Bazsi