[syslog-ng] RE: Re: syslog-ng 2.0rc1 memory usage (Balazs Scheidler)
Martin, David M
David.Martin at anheuser-busch.com
Tue Aug 8 22:56:58 CEST 2006
applied the patch and resident size increased only marginally over a 10 minute
period. much better.
ran through valgrind another 10 minutes, here's the report.
==25356== Memcheck, a memory error detector for x86-linux.
==25356== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al.
==25356== Using valgrind-2.2.0, a program supervision framework for x86-linux.
==25356== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al.
==25356==
==25356== My PID = 25356, parent PID = 1284. Prog and args are:
==25356== ./syslog-ng
==25356== -dv
==25356==
==25356== Valgrind library directory: /usr/lib/valgrind
==25356== Command line
==25356== ./syslog-ng
==25356== -dv
==25356== Startup, with flags:
==25356== --tool=memcheck
==25356== -v
==25356== --leak-check=yes
==25356== --show-reachable=yes
==25356== --log-file=/tmp/debug.txt
==25356== Contents of /proc/version:
==25356== Linux version 2.6.9-34.ELsmp (bhcompile at hs20-bc1-7.build.redhat.com)
(gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 SMP Fri Feb 24 16:54:53 EST
2006
==25356== Reading syms from /opt/sfw/syslogng/2.0rc1/sbin/syslog-ng (0x8048000)
==25356== Reading syms from /lib/ld-2.3.4.so (0x1B8E4000)
==25356== object doesn't have any debug info
==25356== Reading syms from /usr/lib/valgrind/stage2 (0xB0000000)
==25356== Reading syms from /lib/ld-2.3.4.so (0xB1000000)
==25356== object doesn't have any debug info
==25356== Reading syms from /usr/lib/valgrind/vgskin_memcheck.so (0xB7C3E000)
==25356== Reading syms from /lib/tls/libc-2.3.4.so (0xB7E67000)
==25356== object doesn't have any debug info
==25356== Reading syms from /lib/libdl-2.3.4.so (0xB7F92000)
==25356== object doesn't have any debug info
==25356== Reading suppressions file: /usr/lib/valgrind/default.supp
==25356== REDIRECT soname:libc.so.6(__GI___errno_location) to
soname:libpthread.so.0(__errno_location)
==25356== REDIRECT soname:libc.so.6(__errno_location) to
soname:libpthread.so.0(__errno_location)
==25356== REDIRECT soname:libc.so.6(__GI___h_errno_location) to
soname:libpthread.so.0(__h_errno_location)
==25356== REDIRECT soname:libc.so.6(__h_errno_location) to
soname:libpthread.so.0(__h_errno_location)
==25356== REDIRECT soname:libc.so.6(__GI___res_state) to
soname:libpthread.so.0(__res_state)
==25356== REDIRECT soname:libc.so.6(__res_state) to
soname:libpthread.so.0(__res_state)
==25356== REDIRECT soname:libc.so.6(stpcpy) to *vgpreload_memcheck.so*(stpcpy)
==25356== REDIRECT soname:libc.so.6(strnlen) to *vgpreload_memcheck.so*(strnlen)
==25356== REDIRECT soname:ld-linux.so.2(stpcpy) to
*vgpreload_memcheck.so*(stpcpy)
==25356== REDIRECT soname:ld-linux.so.2(strchr) to
*vgpreload_memcheck.so*(strchr)
==25356==
==25356== Reading syms from /usr/lib/valgrind/vg_inject.so (0x1B8FC000)
==25356== Reading syms from /usr/lib/valgrind/vgpreload_memcheck.so (0x1B901000)
==25356== TRANSLATE: 0x1B8F5F70 redirected to 0x1B904100
==25356== Reading syms from /lib/libnsl-2.3.4.so (0xC09000)
==25356== object doesn't have any debug info
==25356== Reading syms from /lib/tls/librt-2.3.4.so (0xDD5000)
==25356== object doesn't have any debug info
==25356== Reading syms from /usr/lib/libglib-2.0.so.0.400.7 (0x101000)
==25356== object doesn't have a symbol table
==25356== object doesn't have any debug info
==25356== Reading syms from /opt/sfw/eventlog/0.2.5/lib/libevtlog.so.0.0.0
(0x1B910000)
==25356== Reading syms from /lib/tls/libc-2.3.4.so (0xA60000)
==25356== object doesn't have any debug info
==25356== Reading syms from /usr/lib/valgrind/libpthread.so (0x1B917000)
==25356== Reading syms from /lib/libdl-2.3.4.so (0xB8C000)
==25356== object doesn't have any debug info
==25356== TRANSLATE: 0xAC4030 redirected to 0x1B9048F8
==25356== TRANSLATE: 0x1B8E47A0 redirected to 0x52BFF040
==25356== TRANSLATE: 0xAC1F60 redirected to 0x1B904E24
==25356== TRANSLATE: 0xAC3D50 redirected to 0x1B905274
==25356== TRANSLATE: 0xA752D0 redirected to 0x1B91EB87
==25356== TRANSLATE: 0xACA370 redirected to 0x1B9046F4
==25356== Reading syms from /lib/libnss_files-2.3.4.so (0x1BA51000)
==25356== object doesn't have any debug info
==25356== TRANSLATE: 0xAC4670 redirected to 0x1B905326
==25356== discard syms at 0x1BA51000-0x1BA5C000 in /lib/libnss_files-2.3.4.so
due to munmap()
==25356==
==25356== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 22 from 2)
--25356--
--25356-- supp: 2 _dl_relocate_object/dl_open_worker
--25356-- supp: 20 dl_relocate_object/dl_main
==25356== malloc/free: in use at exit: 47118 bytes in 370 blocks.
==25356== malloc/free: 535 allocs, 165 frees, 58735 bytes allocated.
==25356==
==25356== searching for pointers to 370 not-freed blocks.
==25360== TRANSLATE: 0xAC9000 redirected to 0x1B9042D0
==25360== Reading syms from /lib/libnss_dns-2.3.4.so (0x1BA49000)
==25360== object doesn't have any debug info
==25360== Reading syms from /lib/libresolv-2.3.4.so (0xC21000)
==25360== object doesn't have any debug info
==25356== checked 2396740 bytes.
==25356==
==25356== 8 bytes in 2 blocks are still reachable in loss record 1 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAF72C8: register_state (in /lib/tls/libc-2.3.4.so)
==25356== by 0xAF765A: create_cd_newstate (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB004C5: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356==
==25356==
==25356== 8 bytes in 2 blocks are still reachable in loss record 2 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAF7439: create_cd_newstate (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB004C5: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356==
==25356==
==25356== 12 bytes in 1 blocks are still reachable in loss record 3 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0x1B9123A7: evt_ctx_tag_hook_add (evtctx.c:126)
==25356== by 0x1B91248B: evt_ctx_init (evtctx.c:151)
==25356== by 0x804C228: msg_init (messages.c:124)
==25356==
==25356==
==25356== 16 bytes in 1 blocks are still reachable in loss record 4 of 19
==25356== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25356== by 0xB8D364: _dlerror_run (in /lib/libdl-2.3.4.so)
==25356== by 0xB8CE3B: dlsym (in /lib/libdl-2.3.4.so)
==25356== by 0x1B91ECA6: __errno_location (vg_libpthread.c:2127)
==25356==
==25356==
==25356== 56 bytes in 14 blocks are still reachable in loss record 5 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAF9D1A: calc_eclosure_iter (in /lib/tls/libc-2.3.4.so)
==25356== by 0xAFF83F: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356==
==25356==
==25356== 56 bytes in 2 blocks are still reachable in loss record 6 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAFF59E: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356== by 0x8051B40: filter_re_compile (filter.c:266)
==25356==
==25356==
==25356== 92 bytes in 1 blocks are still reachable in loss record 7 of 19
==25356== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25356== by 0x1B912410: evt_ctx_init (evtctx.c:142)
==25356== by 0x804C228: msg_init (messages.c:124)
==25356== by 0x804BA3E: main (main.c:389)
==25356==
==25356==
==25356== 104 bytes in 2 blocks are still reachable in loss record 8 of 19
==25356== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25356== by 0xAF73F8: create_cd_newstate (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB004C5: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356==
==25356==
==25356== 112 bytes in 2 blocks are still reachable in loss record 9 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAFEF10: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356== by 0x8051B40: filter_re_compile (filter.c:266)
==25356==
==25356==
==25356== 168 bytes in 2 blocks are still reachable in loss record 10 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAFF5F7: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356== by 0x8051B40: filter_re_compile (filter.c:266)
==25356==
==25356==
==25356== 168 bytes in 2 blocks are still reachable in loss record 11 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAFF5E3: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356== by 0x8051B40: filter_re_compile (filter.c:266)
==25356==
==25356==
==25356== 168 bytes in 2 blocks are still reachable in loss record 12 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xAFF5CF: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356== by 0x8051B40: filter_re_compile (filter.c:266)
==25356==
==25356==
==25356== 288 bytes in 2 blocks are still reachable in loss record 14 of 19
==25356== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25356== by 0xAFEF3A: re_compile_internal (in /lib/tls/libc-2.3.4.so)
==25356== by 0xB012AB: regcomp (in /lib/tls/libc-2.3.4.so)
==25356== by 0x8051B40: filter_re_compile (filter.c:266)
==25356==
==25356==
==25356== 512 bytes in 2 blocks are still reachable in loss record 15 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0xB0123F: regcomp (in /lib/tls/libc-2.3.4.so)
==25356== by 0x8051B40: filter_re_compile (filter.c:266)
==25356== by 0x8051DA5: filter_host_new (filter.c:345)
==25356==
==25356==
==25356== 3608 bytes in 120 blocks are still reachable in loss record 16 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0x1B905373: realloc (vg_replace_malloc.c:189)
==25356== by 0x12AC4B: g_realloc (in /usr/lib/libglib-2.0.so.0.400.7)
==25356== by 0x13C55B: (within /usr/lib/libglib-2.0.so.0.400.7)
==25356==
==25356==
==25356== 4784 bytes in 118 blocks are still reachable in loss record 17 of 19
==25356== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25356== by 0x12ABDF: g_malloc0 (in /usr/lib/libglib-2.0.so.0.400.7)
==25356== by 0x804CA1A: cfg_new (cfg.c:231)
==25356== by 0x804BA53: main (main.c:392)
==25356==
==25356==
==25356== 16438 bytes in 3 blocks are still reachable in loss record 18 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0x804E8EA: yyalloc (cfg-lex.c:1958)
==25356== by 0x804E55B: yyensure_buffer_stack (cfg-lex.c:1693)
==25356== by 0x804DFD0: yyrestart (cfg-lex.c:1467)
==25356==
==25356==
==25356== 20320 bytes in 91 blocks are still reachable in loss record 19 of 19
==25356== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25356== by 0x12AB72: g_malloc (in /usr/lib/libglib-2.0.so.0.400.7)
==25356== by 0x12B6D6: g_mem_chunk_new (in /usr/lib/libglib-2.0.so.0.400.7)
==25356== by 0x130DD7: g_queue_new (in /usr/lib/libglib-2.0.so.0.400.7)
==25356==
==25356== LEAK SUMMARY:
==25356== definitely lost: 0 bytes in 0 blocks.
==25356== possibly lost: 0 bytes in 0 blocks.
==25356== still reachable: 46918 bytes in 369 blocks.
==25356== suppressed: 200 bytes in 1 blocks.
--25356-- TT/TC: 0 tc sectors discarded.
--25356-- 6727 tt_fast misses.
--25356-- translate: new 6283 (108562 -> 1439356; ratio 132:10)
--25356-- discard 124 (1626 -> 21552; ratio 132:10).
--25356-- chainings: 4327 chainings, 3 unchainings.
--25356-- dispatch: 150000 jumps (bb entries); of them 33019 (22%) unchained.
--25356-- 188/7945 major/minor sched events.
--25356-- reg-alloc: 1248 t-req-spill, 266926+8325 orig+spill uis,
--25356-- 32968 total-reg-rank
--25356-- sanity: 99 cheap, 4 expensive checks.
--25356-- ccalls: 27187 C calls, 55% saves+restores avoided (89446 bytes)
--25356-- 36049 args, avg 0.87 setup instrs each (8718 bytes)
--25356-- 0% clear the stack (81192 bytes)
--25356-- 10763 retvals, 32% of reg-reg movs avoided (6762 bytes)
==25360== TRANSLATE: 0xB37F10 redirected to 0x1B91EE2A
==25360== TRANSLATE: 0xB3C050 redirected to 0x1B91ED13
==25360== discard syms at 0x1BA51000-0x1BA5C000 in /lib/libnss_files-2.3.4.so
due to munmap()
==25360== discard syms at 0x1BA49000-0x1BA4F000 in /lib/libnss_dns-2.3.4.so due
to munmap()
==25360== discard syms at 0xC21000-0xC34000 in /lib/libresolv-2.3.4.so due to
munmap()
==25360==
==25360== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 25 from 2)
--25360--
--25360-- supp: 5 _dl_relocate_object/dl_open_worker
--25360-- supp: 20 dl_relocate_object/dl_main
==25360== malloc/free: in use at exit: 38779 bytes in 173 blocks.
==25360== malloc/free: 12356936 allocs, 12356763 frees, 294589976 bytes
allocated.
==25360==
==25360== searching for pointers to 173 not-freed blocks.
==25360== checked 2394928 bytes.
==25360==
==25360== 16 bytes in 1 blocks are still reachable in loss record 1 of 8
==25360== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25360== by 0xB8D364: _dlerror_run (in /lib/libdl-2.3.4.so)
==25360== by 0xB8CE3B: dlsym (in /lib/libdl-2.3.4.so)
==25360== by 0x1B91ECA6: __errno_location (vg_libpthread.c:2127)
==25360==
==25360==
==25360== 108 bytes in 1 blocks are definitely lost in loss record 2 of 8
==25360== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25360== by 0x12ABDF: g_malloc0 (in /usr/lib/libglib-2.0.so.0.400.7)
==25360== by 0x80530FD: log_msg_new (logmsg.c:545)
==25360== by 0x804C0A7: msg_send_internal_message (messages.c:57)
==25360==
==25360==
==25360== 128 bytes in 1 blocks are still reachable in loss record 3 of 8
==25360== at 0x1B9053EE: realloc (vg_replace_malloc.c:197)
==25360== by 0x12AC4B: g_realloc (in /usr/lib/libglib-2.0.so.0.400.7)
==25360== by 0x13C55B: (within /usr/lib/libglib-2.0.so.0.400.7)
==25360== by 0x13CACD: g_string_insert_len (in
/usr/lib/libglib-2.0.so.0.400.7)
==25360==
==25360==
==25360== 3272 bytes in 45 blocks are still reachable in loss record 5 of 8
==25360== at 0x1B905301: calloc (vg_replace_malloc.c:176)
==25360== by 0x12ABDF: g_malloc0 (in /usr/lib/libglib-2.0.so.0.400.7)
==25360== by 0x80530FD: log_msg_new (logmsg.c:545)
==25360== by 0x804C0A7: msg_send_internal_message (messages.c:57)
==25360==
==25360==
==25360== 3984 bytes in 84 blocks are still reachable in loss record 6 of 8
==25360== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25360== by 0x1B905373: realloc (vg_replace_malloc.c:189)
==25360== by 0x12AC4B: g_realloc (in /usr/lib/libglib-2.0.so.0.400.7)
==25360== by 0x13C55B: (within /usr/lib/libglib-2.0.so.0.400.7)
==25360==
==25360==
==25360== 14633 bytes in 37 blocks are still reachable in loss record 7 of 8
==25360== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25360== by 0x12AB72: g_malloc (in /usr/lib/libglib-2.0.so.0.400.7)
==25360== by 0x12B6D6: g_mem_chunk_new (in /usr/lib/libglib-2.0.so.0.400.7)
==25360== by 0x130DD7: g_queue_new (in /usr/lib/libglib-2.0.so.0.400.7)
==25360==
==25360==
==25360== 16438 bytes in 3 blocks are still reachable in loss record 8 of 8
==25360== at 0x1B904984: malloc (vg_replace_malloc.c:131)
==25360== by 0x804E8EA: yyalloc (cfg-lex.c:1958)
==25360== by 0x804E55B: yyensure_buffer_stack (cfg-lex.c:1693)
==25360== by 0x804DFD0: yyrestart (cfg-lex.c:1467)
==25360==
==25360== LEAK SUMMARY:
==25360== definitely lost: 108 bytes in 1 blocks.
==25360== possibly lost: 0 bytes in 0 blocks.
==25360== still reachable: 38471 bytes in 171 blocks.
==25360== suppressed: 200 bytes in 1 blocks.
--25360-- TT/TC: 0 tc sectors discarded.
--25360-- 3473105 tt_fast misses.
--25360-- translate: new 10152 (171521 -> 2315890; ratio 135:10)
--25360-- discard 687 (11209 -> 143352; ratio 127:10).
--25360-- chainings: 7759 chainings, 3 unchainings.
--25360-- dispatch: 200000 jumps (bb entries); of them 337925145 (168962%)
unchained.
--25360-- 1207539/32139357 major/minor sched events.
--25360-- reg-alloc: 1915 t-req-spill, 428681+13190 orig+spill uis,
--25360-- 52732 total-reg-rank
--25360-- sanity: 641240 cheap, 25650 expensive checks.
--25360-- ccalls: 44737 C calls, 56% saves+restores avoided (149268 bytes)
--25360-- 58904 args, avg 0.88 setup instrs each (13864 bytes)
--25360-- 0% clear the stack (133704 bytes)
--25360-- 17609 retvals, 33% of reg-reg movs avoided (11332 bytes)
------------------------------
Message: 8
Date: Tue, 08 Aug 2006 22:02:03 +0200
From: Balazs Scheidler <bazsi at balabit.hu>
Subject: Re: [syslog-ng] Re: syslog-ng 2.0rc1 memory usage
To: Syslog-ng users' and developers' mailing list
<syslog-ng at lists.balabit.hu>
Message-ID: <1155067324.11773.2.camel at bzorp.balabit>
Content-Type: text/plain; charset="us-ascii"
On Mon, 2006-08-07 at 19:11 -0500, Martin, David M wrote:
> I ran valgrind with these options for 10 minutes:
>
> valgrind --tool=memcheck -v --leak-check=yes --show-reachable=yes
> --log-file=/tmp/debug.txt ./syslog-ng -dv
Attached my latest set of leak fixes, please check if it fixes your
problem. I've also committed it, so it should be available in tomorrow's
snapshot.
Thanks again for helping me to debug this problem.
--
Bazsi
-------------- next part --------------
* added files
{arch}/syslog-ng/syslog-ng--mainline/syslog-ng--mainline--2.0/devel at balabit.hu--
other-1/patch-log/patch-73
* modified files
--- orig/ChangeLog
+++ mod/ChangeLog
@@ -2,6 +2,52 @@
# arch-tag:
automatic-ChangeLog--devel at balabit.hu--other-1/syslog-ng--mainline--2.0
#
The information transmitted (including attachments) is
covered by the Electronic Communications Privacy Act,
18 U.S.C. 2510-2521, is intended only for the person(s) or
entity/entities to which it is addressed and may contain
confidential and/or privileged material. Any review,
retransmission, dissemination or other use of, or taking
of any action in reliance upon, this information by persons
or entities other than the intended recipient(s) is prohibited.
If you received this in error, please contact the sender and
delete the material from any computer.
More information about the syslog-ng
mailing list