[Bug 208] New: Memory leak when enabling debug logging
https://bugzilla.balabit.com/show_bug.cgi?id=208 Summary: Memory leak when enabling debug logging Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: koldaevav@gmail.com Type of the Report: bug Estimated Hours: 0.0 syslog-ng, all default configuration root@test-syslog-02:~# uname -a Linux test-syslog-02 2.6.32-44-server #98-Ubuntu SMP Mon Sep 24 17:41:33 UTC 2012 x86_64 GNU/Linux root@test-syslog-02:~# cat /etc/issue Ubuntu 10.04.4 LTS \n \l root@test-syslog-02:~# dpkg -l | grep syslog ii libsyslog-ng-3.3.6 3.3.6-1~mhp1~lucid Next generation system logging daemon (private librar ii libsyslog-ng-3.3.6.91 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (private librar ii syslog-ng 3.3.6.91-1~mhp1 Next generation system logging daemon (metapackage) ii syslog-ng-core 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (core) ii syslog-ng-mod-json 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (JSON plugin) ii syslog-ng-mod-mongodb 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (MongoDB plugin ii syslog-ng-mod-sql 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (SQL plugin) root@test-syslog-02:~# syslog-ng-ctl debug --set=on && strace -f -c -p `pgrep -f '/usr/sbin/syslog-ng'` Process 30701 attached with 2 threads - interrupt to quit Process 30704 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.150000 16667 9 epoll_wait 0.00 0.000000 0 2 read 0.00 0.000000 0 5 write 0.00 0.000000 0 2 close 0.00 0.000000 0 22129 brk 0.00 0.000000 0 1 madvise 0.00 0.000000 0 1 accept 0.00 0.000000 0 1 setsockopt 0.00 0.000000 0 4 fcntl 0.00 0.000000 0 3 futex 0.00 0.000000 0 4 epoll_ctl ------ ----------- ----------- --------- --------- ---------------- 100.00 0.150000 22161 total It uses all available memory and swap, and being killed by oom_killer: Oct 26 13:44:24 test-syslog-02 kernel: : [107460.493074] Out of memory: kill process 30701 (syslog-ng) score 186140 or a child Oct 26 13:44:24 test-syslog-02 kernel: : [107460.648697] Killed process 30701 (syslog-ng) root@test-syslog-02:~# syslog-ng-ctl debug --set=on; strace -f -o ./syslog-ng.strace.log -p `pgrep -f '/usr/sbin/syslog-ng'` root@test-syslog-02:~# grep -v brk ./syslog-ng.strace.log 31364 epoll_wait(18, <unfinished ...> 31365 epoll_wait(21, <unfinished ...> 31365 <... epoll_wait resumed> {}, 1, 3194) = 0 31364 <... epoll_wait resumed> {}, 1, 3193) = 0 31365 epoll_ctl(21, EPOLL_CTL_DEL, 22, {0, {u32=3357993160, u64=139959162300616}} <unfinished ...> 31364 futex(0x7f4ac813fc20, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> 31365 <... epoll_ctl resumed> ) = 0 31365 close(22) = 0 31365 futex(0x7f4ac813fc20, FUTEX_WAKE_PRIVATE, 1) = 1 31364 <... futex resumed> ) = 0 31365 close(21 <unfinished ...> 31364 epoll_ctl(18, EPOLL_CTL_DEL, 19, {0, {u32=3357942664, u64=139959162250120}} <unfinished ...> 31365 <... close resumed> ) = 0 31364 <... epoll_ctl resumed> ) = 0 31365 write(6, "\1\0\0\0\0\0\0\0", 8 <unfinished ...> 31364 close(19 <unfinished ...> 31365 <... write resumed> ) = 8 31364 <... close resumed> ) = 0 31365 madvise(0x7f4ac1b9c000, 8359936, MADV_DONTNEED <unfinished ...> 31364 futex(0x7f4ac813fc20, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 31365 <... madvise resumed> ) = 0 31364 <... futex resumed> ) = 0 31365 _exit(0) = ? 31364 close(18) = 0 31364 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 31364 madvise(0x7f4ac239d000, 8359936, MADV_DONTNEED) = 0 31364 _exit(0) = ? 31362 +++ killed by SIGKILL +++ I'll be happy to provide any additional info you need. I'm not able to debug my syslog-ng problems because if this bug. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |3.3.7 CC| |algernon@balabit.hu AssignedTo|bazsi@balabit.hu |algernon@balabit.hu -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #1 from Gergely Nagy <algernon@balabit.hu> 2012-10-27 11:40:59 --- Ouch. This is trivially reproducible, but only happens when setting the debug level with syslog-ng-ctl, as far as I see - when running with -d from the start, the leak does not happen. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #2 from Anton <koldaevav@gmail.com> 2012-10-27 11:59:50 --- (In reply to comment #1)
Ouch. This is trivially reproducible, but only happens when setting the debug level with syslog-ng-ctl, as far as I see - when running with -d from the start, the leak does not happen.
Right. But when I'm running syslog-ng with '-d' it handles only 5-10% of all logs(in my custom configuration). I'm not sure if it's because of my current configuration but everything works great without '-d'. I would appreciate if somebody could try to generate some load when running syslog-ng with '-d' option. So I have no change to debug my installation and would be really happy if developers are able to fix memory leak bug. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #3 from Gergely Nagy <algernon@balabit.hu> 2012-10-27 12:13:19 --- (In reply to comment #2)
(In reply to comment #1)
Ouch. This is trivially reproducible, but only happens when setting the debug level with syslog-ng-ctl, as far as I see - when running with -d from the start, the leak does not happen.
Right. But when I'm running syslog-ng with '-d' it handles only 5-10% of all logs(in my custom configuration). I'm not sure if it's because of my current configuration but everything works great without '-d'. I would appreciate if somebody could try to generate some load when running syslog-ng with '-d' option.
I didn't mean to imply you should run with -d - I was merely observing what happens.
So I have no change to debug my installation and would be really happy if developers are able to fix memory leak bug.
I'm working on it. Got as far as to produce a minimal config the problem happens with: @version: 3.3 source s_internal { internal(); }; filter f_debug { level(debug); }; destination d_null {}; log { source(s_internal); filter(f_debug); destination(d_null); }; If either internal() or the filter is removed, the leak does not happen. From what I see from the core dump that happens when syslog-ng reaches its memory limit, the filter-generated debug messages are the things that are leaking rapidly. Looks like we get into an infinite loop processing our own debug messages, and producing a dozen new ones for each. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #4 from Gergely Nagy <algernon@balabit.hu> 2012-10-27 13:03:01 --- Hrm. This is - despite the looks of it - not a memory leak, but a feedback loop: enabling debug messages via syslog-ng ctl will trigger an internal message, which will run through a bunch of filters, each of which triggers more internal messages, which will go through the same filters, tiggering even more messages - until we run out of memory. There are a few possible solutions I can see to remedy this situation: - Internal messages should not generate further internal messages: this has the horrible disadvantage that this makes it impossible to debug where internal messages and up. So... no go. (It is also awkward to implement) - Change the config so that internal messages are routed to a separate file, all of them, without filtering. This avoids the problematic situation, but results in having a completely separate log file for syslog-ng internal messages. - Internal messages generated by internal messages should not generate further internal messages: this is a variant of the first, but it allows for one set of child messages, making it possible to debug the flow of internal messages too. The downside is that this is the hardest to implement. I would suggest going with the second option, at least until a better solution is found. This would mean removing the internal() statement from the s_src source, and creating a new source, destination & logpath trio: source s_internal { internal(); } destination d_internal { file("/var/log/syslog-ng.log"); } log { source(s_internal); destination(d_internal); flags(final); } I'll do that for the next version of the Debian/Ubuntu packages. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Balazs Scheidler <bazsi@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bazsi@balabit.hu --- Comment #5 from Balazs Scheidler <bazsi@balabit.hu> 2012-10-28 10:32:55 --- there's a feedback loop prevention stuff in syslog-ng, it just doesn't kick in. patch follows shortly. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #6 from Balazs Scheidler <bazsi@balabit.hu> 2012-10-28 11:37:59 --- My mess detector detected some mess, so I intend to do some refactoring in this area, but until that's ready, this simple patch should be straightforward to backport to 3.3 and seems to fix the problem for me. commit e756e45f9d51c7f301ba411fc76d25d532865a7a Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Oct 28 10:25:16 2012 +0100 messages.h: check positive feedback loop for debug/trace messages too Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #7 from Balazs Scheidler <bazsi@balabit.hu> 2012-10-28 13:44:13 --- the refactorization stuff is also on 3.4/master now. commit f26196618d4407f3bb574ea991b36c920fb09c19 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Oct 28 13:04:41 2012 +0100 messages: only allow a single level of recursions The code as originally written has permitted a configurable number of recursions to detect the positive feedback loop of internal messages. This feedback loop may happen when an internal() message triggers another internal() which in turn again and so on, effectively eating all the available system memory. Since we've used a single level of recursions anyway, this patch simplifies the related code and frees up 6 bits in the LogMessage structure. It also adds some more information into the suppression message, so it can be easier to understand. Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> commit d7fd9a401da09ab7d638257293c6fee0f9b8b35d Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Oct 28 13:04:35 2012 +0100 messages: remove redirect_to_syslog functionality it was completely unused. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|ASSIGNED |RESOLVED --- Comment #8 from Gergely Nagy <algernon@balabit.hu> 2012-10-28 15:38:37 --- (In reply to comment #6)
My mess detector detected some mess, so I intend to do some refactoring in this area, but until that's ready, this simple patch should be straightforward to backport to 3.3 and seems to fix the problem for me.
A straight cherry-pick worked, and the problem's gone. Thanks a lot! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #9 from Anton <koldaevav@gmail.com> 2012-10-28 15:44:32 --- (In reply to comment #8)
(In reply to comment #6)
My mess detector detected some mess, so I intend to do some refactoring in this area, but until that's ready, this simple patch should be straightforward to backport to 3.3 and seems to fix the problem for me.
A straight cherry-pick worked, and the problem's gone. Thanks a lot!
Great, thanks for such a quick fix. Any chance to see updated syslog-ng-3.3 deb-package? -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #10 from Gergely Nagy <algernon@balabit.hu> 2012-10-28 16:02:56 --- (In reply to comment #9)
Great, thanks for such a quick fix.
Any chance to see updated syslog-ng-3.3 deb-package?
There will be 3.3.7 packages on Oct 31, I'm not in a position to build snapshot packages before that. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #11 from Anton <koldaevav@gmail.com> 2012-10-28 19:29:59 --- (In reply to comment #10)
(In reply to comment #9)
Great, thanks for such a quick fix.
Any chance to see updated syslog-ng-3.3 deb-package?
There will be 3.3.7 packages on Oct 31, I'm not in a position to build snapshot packages before that.
Could you please point me to the actual repo so I can start using the fix right now? Looks like github repo is a bit outdated -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #12 from Gergely Nagy <algernon@balabit.hu> 2012-10-28 20:55:15 --- (In reply to comment #11)
(In reply to comment #10)
(In reply to comment #9)
Great, thanks for such a quick fix.
Any chance to see updated syslog-ng-3.3 deb-package?
There will be 3.3.7 packages on Oct 31, I'm not in a position to build snapshot packages before that.
Could you please point me to the actual repo so I can start using the fix right now? Looks like github repo is a bit outdated
https://github.com/balabit/syslog-ng-3.3 The master branch has the fix. If you want to build a Debian package, apt-get source syslog-ng, and copy the debian/ dir over, that should do the trick. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #13 from Anton <koldaevav@gmail.com> 2012-10-28 20:59:30 --- (In reply to comment #12)
(In reply to comment #11)
(In reply to comment #10)
(In reply to comment #9)
Great, thanks for such a quick fix.
Any chance to see updated syslog-ng-3.3 deb-package?
There will be 3.3.7 packages on Oct 31, I'm not in a position to build snapshot packages before that.
Could you please point me to the actual repo so I can start using the fix right now? Looks like github repo is a bit outdated
https://github.com/balabit/syslog-ng-3.3
The master branch has the fix. If you want to build a Debian package, apt-get source syslog-ng, and copy the debian/ dir over, that should do the trick.
Thanks!. I was looking at https://github.com/bazsi/syslog-ng-3.3 before. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Anton <koldaevav@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|FIXED | Status|RESOLVED |REOPENED --- Comment #14 from Anton <koldaevav@gmail.com> 2012-10-31 14:33:19 --- Unfortunately I'm seeing almost the same problem even with updated version(3.3.7). Looks like I'm not able to reproduce it this time on my test server but in production I see syslog-ng consumes all the memory again. It stops writing any logs right after enabling debug. The process of memory consumption about 3x slower than without this bugfix. Oct 31 11:37:27 syslog-01 syslog-ng[28181]: Verbosity changed; type='DEBUG', on='1' Oct 31 11:37:27 syslog-01 syslog-ng[28181]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='2' Oct 31 11:39:23 syslog-01 syslog-ng[28181]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='2' Strace: ... 30135 mprotect(0x7f6025b09000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30126 mprotect(0x7f60013aa000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30131 mprotect(0x7f6010db5000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30117 <... futex resumed> ) = 0 30135 <... mprotect resumed> ) = 0 30131 <... mprotect resumed> ) = 0 30126 <... mprotect resumed> ) = 0 30135 futex(0x7f602df81148, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> 30131 futex(0x7f602df81148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 30135 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) 30131 <... futex resumed> ) = 0 30135 futex(0x7f602df81148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 30126 mprotect(0x7f60013ab000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30135 <... futex resumed> ) = 0 30131 mprotect(0x7f6010db6000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30117 mprotect(0x7f60156a3000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30135 mprotect(0x7f6025b0a000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30126 <... mprotect resumed> ) = 0 30131 <... mprotect resumed> ) = 0 30135 <... mprotect resumed> ) = 0 .... 30117 futex(0x7f602df81148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 30131 mprotect(0x7f6010e6f000, 4096, PROT_READ|PROT_WRITE <unfinished ...> 30117 <... futex resumed> ) = 0 30131 <... mprotect resumed> ) = 0 30117 mprotect(0x7f60156ba000, 4096, PROT_READ|PROT_WRITE) = 0 30131 mprotect(0x7f6010e70000, 4096, PROT_READ|PROT_WRITE) = 0 30117 mprotect(0x7f60156bb000, 4096, PROT_READ|PROT_WRITE) = 0 30131 mprotect(0x7f6010e71000, 4096, PROT_READ|PROT_WRITE) = 0 30117 mprotect(0x7f60156bc000, 4096, PROT_READ|PROT_WRITE) = 0 [skipped tons of mprotect] Memory consumption: % for i in {1..1000}; do date; free -m; sleep 2; done Wed Oct 31 13:24:35 UTC 2012 total used free shared buffers cached Mem: 7990 7955 34 0 181 6640 -/+ buffers/cache: 1133 6857 Swap: 2047 135 1912 Wed Oct 31 13:24:37 UTC 2012 total used free shared buffers cached Mem: 7990 7942 47 0 180 6521 -/+ buffers/cache: 1240 6749 Swap: 2047 135 1912 Wed Oct 31 13:24:39 UTC 2012 total used free shared buffers cached Mem: 7990 7943 46 0 179 6398 -/+ buffers/cache: 1366 6623 Swap: 2047 135 1912 Wed Oct 31 13:24:41 UTC 2012 total used free shared buffers cached Mem: 7990 7948 41 0 178 6301 -/+ buffers/cache: 1467 6522 Swap: 2047 135 1912 Wed Oct 31 13:24:43 UTC 2012 total used free shared buffers cached Mem: 7990 7944 45 0 178 6207 -/+ buffers/cache: 1559 6430 Swap: 2047 135 1912 Wed Oct 31 13:24:45 UTC 2012 total used free shared buffers cached Mem: 7990 7946 43 0 177 6112 -/+ buffers/cache: 1656 6333 Swap: 2047 135 1912 Wed Oct 31 13:24:47 UTC 2012 total used free shared buffers cached Mem: 7990 7947 42 0 176 5993 -/+ buffers/cache: 1777 6213 Swap: 2047 135 1912 Wed Oct 31 13:24:49 UTC 2012 total used free shared buffers cached Mem: 7990 7955 35 0 174 5831 -/+ buffers/cache: 1948 6041 Swap: 2047 135 1912 Wed Oct 31 13:24:52 UTC 2012 total used free shared buffers cached Mem: 7990 7950 39 0 173 5707 -/+ buffers/cache: 2068 5921 Swap: 2047 135 1912 Wed Oct 31 13:24:54 UTC 2012 total used free shared buffers cached Mem: 7990 7948 41 0 172 5589 -/+ buffers/cache: 2186 5803 Swap: 2047 135 1912 You can see my production configuration here: https://gist.github.com/c4688693f8b9851c6a41 I'll be happy to provide any additional info you need. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 --- Comment #15 from Gergely Nagy <algernon@balabit.hu> 2012-10-31 14:45:58 --- (In reply to comment #14)
Unfortunately I'm seeing almost the same problem even with updated version(3.3.7). Looks like I'm not able to reproduce it this time on my test server but in production I see syslog-ng consumes all the memory again.
Mhhmm. I can reproduce it too, but I won't have time to investigate further for a few days. In the meanwhile, I'd suggest pulling out the internal() source into its own source, and add it to a logpath that does not go through filters. That should work around the issue until we can fix it properly. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|3.3.7 |3.3.8 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Anton <koldaevav@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|3.3.8 |3.3.7 --- Comment #16 from Anton <koldaevav@gmail.com> 2012-10-31 16:28:38 --- (In reply to comment #15)
(In reply to comment #14)
Unfortunately I'm seeing almost the same problem even with updated version(3.3.7). Looks like I'm not able to reproduce it this time on my test server but in production I see syslog-ng consumes all the memory again.
Mhhmm. I can reproduce it too, but I won't have time to investigate further for a few days. In the meanwhile, I'd suggest pulling out the internal() source into its own source, and add it to a logpath that does not go through filters. That should work around the issue until we can fix it properly.
I just did as you say: source s_local_internal { internal(); }; block root b_general_local(type(), flags()) { destination d_`type`_internal { file("/u/logs/syslog/`type`.all.log", owner(root), group(app), perm(0644)); }; log { source(s_`type`_internal); destination(d_`type`_internal); flags(`flags`); }; }; b_general_local(type(local)) nothing else logging to s_local_internal And I'm getting just the same problem with memory consumption. Strace: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 68.74 54.859307 223916 245 epoll_wait 25.83 20.617957 20 1016964 333603 futex 2.19 1.748952 6 298470 mprotect 1.38 1.104910 1044 1058 fcntl 1.14 0.910307 4484 203 epoll_ctl 0.38 0.306601 581 528 18 setsockopt 0.24 0.193717 59 3304 write 0.05 0.037166 70 534 87 read 0.03 0.019979 377 53 close 0.01 0.009524 4 2170 writev 0.00 0.000822 37 22 fstat 0.00 0.000474 0 2170 lseek 0.00 0.000058 0 623 stat 0.00 0.000050 0 915 brk 0.00 0.000025 1 42 mmap 0.00 0.000000 0 22 open 0.00 0.000000 0 42 poll 0.00 0.000000 0 32 munmap 0.00 0.000000 0 21 ioctl 0.00 0.000000 0 2 madvise 0.00 0.000000 0 21 socket 0.00 0.000000 0 21 connect 0.00 0.000000 0 21 sendto 0.00 0.000000 0 89 recvfrom ------ ----------- ----------- --------- --------- ---------------- 100.00 79.809849 1327572 333708 total -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Anton <koldaevav@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|3.3.7 |3.3.8 --- Comment #17 from Anton <koldaevav@gmail.com> 2012-10-31 16:32:17 --- oops, changed target milestone by mistake -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |ASSIGNED --- Comment #18 from Gergely Nagy <algernon@balabit.hu> 2013-01-15 12:18:53 --- (In reply to comment #15)
(In reply to comment #14)
Unfortunately I'm seeing almost the same problem even with updated version(3.3.7). Looks like I'm not able to reproduce it this time on my test server but in production I see syslog-ng consumes all the memory again.
Mhhmm. I can reproduce it too, but I won't have time to investigate further for a few days. In the meanwhile, I'd suggest pulling out the internal() source into its own source, and add it to a logpath that does not go through filters. That should work around the issue until we can fix it properly.
I haven't been able to reproduce it since (only once, but then a few days later, I couldn't make it OOM again :/), but I'll give it another go tomorrow. Are you still experiencing the issue with current 3.3 git master? -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=208 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|3.3.8 |--- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla＠bugzilla.balabit.com