[syslog-ng] syslog-ng never frees RAM
Jose Angel Santiago
jasantiago at stratio.com
Mon Sep 10 06:25:25 UTC 2018
Hi,
We've tested these 2 fixes along with the previous one during this weekend
in our demo environment and now the RAM usage seems to be pretty fair, so I
guess fixes are working great!
Thanks for your quick & effective response.
2018-09-06 16:45 GMT+02:00 Nemes, Antal <antal.nemes at oneidentity.com>:
> Hi,
>
> We found another leak related to monitor-method(poll). When the tracked
> directory is deleted, glib allocates a GError structure for "Error opening
> directory \'...\': No such file or directory", that was not freed by
> syslog-ng. This can cause considerable memory leak when high number of
> directories tracked and deleted. We created a pull request to fix it: here
> you can track the progress: https://github.com/balabit/syslog-ng/pull/2267
>
> Br,
> Antal
>
> On Thu, Sep 6, 2018 at 1:15 PM Jose Angel Santiago <jasantiago at stratio.com>
> wrote:
>
>> Hi,
>>
>> I had to restart syslog-ng process because it was affecting other
>> services in the same host.
>>
>> I've checked another host with was running with the fix too, it has 23000
>> files monitored and 38700 folders under base-dir, and now it's starting to
>> eat the RAM, the cleaning process will be executed in about an hour (I'm
>> sorry I can't force to do it before)
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 96 root 20 0 *5.609g 4.274g* 0.006g R *101.3* *6.8 * 1263:46
>> syslog-ng
>>
>> Trace of pmap:
>>
>> 96: /syslog-ng/exec/sbin/syslog-ng -F --no-caps -f
>> /etc/syslog-ng/syslog-ng.conf -p /var/syslog-ng/syslog-ng.pid -R
>> /var/syslog-ng/syslog-ng.persist -c /var/syslog-ng/syslog-ng.ctl
>> 0000000000400000 12K r-x-- syslog-ng
>> 0000000000602000 4K r---- syslog-ng
>> 0000000000603000 4K rw--- syslog-ng
>> 00000000015bc000 5689360K rw--- [ anon ]
>> 00007ff9c8000000 19588K rw--- [ anon ]
>> 00007ff9c9321000 45948K ----- [ anon ]
>> 00007ff9cc000000 4288K rw--- [ anon ]
>> 00007ff9cc430000 61248K ----- [ anon ]
>> 00007ff9d0000000 26844K rw--- [ anon ]
>> 00007ff9d1a37000 38692K ----- [ anon ]
>> 00007ff9d4000000 10280K rw--- [ anon ]
>> 00007ff9d4a0a000 55256K ----- [ anon ]
>> 00007ff9dc000000 64604K rw--- [ anon ]
>> 00007ff9dff17000 932K ----- [ anon ]
>> 00007ff9e0000000 65508K rw--- [ anon ]
>> 00007ff9e3ff9000 28K ----- [ anon ]
>> 00007ff9e8000000 4480K rw--- [ anon ]
>> 00007ff9e8460000 61056K ----- [ anon ]
>> 00007ff9f0000000 65488K rw--- [ anon ]
>> 00007ff9f3ff4000 48K ----- [ anon ]
>> 00007ff9f8000000 14716K rw--- [ anon ]
>> 00007ff9f8e5f000 50820K ----- [ anon ]
>> 00007ffa00000000 18252K rw--- [ anon ]
>> 00007ffa011d3000 47284K ----- [ anon ]
>> 00007ffa08000000 65536K rw--- [ anon ]
>> 00007ffa0c000000 30460K rw--- [ anon ]
>> 00007ffa0ddbf000 35076K ----- [ anon ]
>> 00007ffa10000000 65484K rw--- [ anon ]
>> 00007ffa13ff3000 52K ----- [ anon ]
>> 00007ffa167fd000 4K ----- [ anon ]
>> 00007ffa167fe000 8192K rw--- [ anon ]
>> 00007ffa18000000 7920K rw--- [ anon ]
>> 00007ffa187bc000 57616K ----- [ anon ]
>> 00007ffa1cffa000 4K ----- [ anon ]
>> 00007ffa1cffb000 8192K rw--- [ anon ]
>> 00007ffa1f7ff000 4K ----- [ anon ]
>> 00007ffa1f800000 8192K rw--- [ anon ]
>> 00007ffa20000000 65536K rw--- [ anon ]
>> 00007ffa24000000 14764K rw--- [ anon ]
>> 00007ffa24e6b000 50772K ----- [ anon ]
>> 00007ffa28000000 24348K rw--- [ anon ]
>> 00007ffa297c7000 41188K ----- [ anon ]
>> 00007ffa2c000000 40916K rw--- [ anon ]
>> 00007ffa2e7f5000 24620K ----- [ anon ]
>> 00007ffa30000000 17684K rw--- [ anon ]
>> 00007ffa31145000 47852K ----- [ anon ]
>> 00007ffa34000000 10988K rw--- [ anon ]
>> 00007ffa34abb000 54548K ----- [ anon ]
>> 00007ffa38000000 20464K rw--- [ anon ]
>> 00007ffa393fc000 45072K ----- [ anon ]
>> 00007ffa3c316000 4K ----- [ anon ]
>> 00007ffa3c317000 8192K rw--- [ anon ]
>> 00007ffa3ea78000 4K ----- [ anon ]
>> 00007ffa3ea79000 8192K rw--- [ anon ]
>> 00007ffa3fbee000 7360K rw-s- syslog-ng.persist
>> 00007ffa4031e000 92K r-x-- libresolv-2.23.so
>> 00007ffa40335000 2048K ----- libresolv-2.23.so
>> 00007ffa40535000 4K r---- libresolv-2.23.so
>> 00007ffa40536000 4K rw--- libresolv-2.23.so
>> 00007ffa40537000 8K rw--- [ anon ]
>> 00007ffa40539000 20K r-x-- libnss_dns-2.23.so
>> 00007ffa4053e000 2048K ----- libnss_dns-2.23.so
>> 00007ffa4073e000 4K r---- libnss_dns-2.23.so
>> 00007ffa4073f000 4K rw--- libnss_dns-2.23.so
>> 00007ffa40740000 3328K rw--- [ anon ]
>> 00007ffa40ac0000 256K rw--- [ anon ]
>> 00007ffa40b40000 512K rw--- [ anon ]
>> 00007ffa40c00000 256K rw--- [ anon ]
>> 00007ffa40c6c000 256K rw--- [ anon ]
>> 00007ffa40d6c000 44K r-x-- libsyslogformat.so
>> 00007ffa40d77000 2044K ----- libsyslogformat.so
>> 00007ffa40f76000 4K r---- libsyslogformat.so
>> 00007ffa40f77000 4K rw--- libsyslogformat.so
>> 00007ffa40f78000 4612K rw--- [ anon ]
>> 00007ffa413f9000 24K r-x-- _hashlib.x86_64-linux-gnu.so
>> 00007ffa413ff000 2044K ----- _hashlib.x86_64-linux-gnu.so
>> 00007ffa415fe000 4K r---- _hashlib.x86_64-linux-gnu.so
>> 00007ffa415ff000 4K rw--- _hashlib.x86_64-linux-gnu.so
>> 00007ffa41600000 256K rw--- [ anon ]
>> 00007ffa41640000 48K r-x-- _json.x86_64-linux-gnu.so
>> 00007ffa4164c000 2044K ----- _json.x86_64-linux-gnu.so
>> 00007ffa4184b000 4K r---- _json.x86_64-linux-gnu.so
>> 00007ffa4184c000 4K rw--- _json.x86_64-linux-gnu.so
>> 00007ffa4184d000 84K r-x-- _ssl.x86_64-linux-gnu.so
>> 00007ffa41862000 2044K ----- _ssl.x86_64-linux-gnu.so
>> 00007ffa41a61000 4K r---- _ssl.x86_64-linux-gnu.so
>> 00007ffa41a62000 16K rw--- _ssl.x86_64-linux-gnu.so
>> 00007ffa41a66000 768K rw--- [ anon ]
>> 00007ffa41b57000 256K rw--- [ anon ]
>> 00007ffa41b97000 1056K r-x-- libm-2.23.so
>> 00007ffa41c9f000 2044K ----- libm-2.23.so
>> 00007ffa41e9e000 4K r---- libm-2.23.so
>> 00007ffa41e9f000 4K rw--- libm-2.23.so
>> 00007ffa41ea0000 8K r-x-- libutil-2.23.so
>> 00007ffa41ea2000 2044K ----- libutil-2.23.so
>> 00007ffa420a1000 4K r---- libutil-2.23.so
>> 00007ffa420a2000 4K rw--- libutil-2.23.so
>> 00007ffa420a3000 100K r-x-- libz.so.1.2.8
>> 00007ffa420bc000 2044K ----- libz.so.1.2.8
>> 00007ffa422bb000 4K r---- libz.so.1.2.8
>> 00007ffa422bc000 4K rw--- libz.so.1.2.8
>> 00007ffa422bd000 3016K r-x-- libpython2.7.so.1.0
>> 00007ffa425af000 2048K ----- libpython2.7.so.1.0
>> 00007ffa427af000 8K r---- libpython2.7.so.1.0
>> 00007ffa427b1000 476K rw--- libpython2.7.so.1.0
>> 00007ffa42828000 140K rw--- [ anon ]
>> 00007ffa4284b000 72K r-x-- libmod-python.so
>> 00007ffa4285d000 2044K ----- libmod-python.so
>> 00007ffa42a5c000 4K r---- libmod-python.so
>> 00007ffa42a5d000 4K rw--- libmod-python.so
>> 00007ffa42a5e000 4K rw--- [ anon ]
>> 00007ffa42a5f000 32K r-x-- libwrap.so.0.7.6
>> 00007ffa42a67000 2044K ----- libwrap.so.0.7.6
>> 00007ffa42c66000 4K r---- libwrap.so.0.7.6
>> 00007ffa42c67000 4K rw--- libwrap.so.0.7.6
>> 00007ffa42c68000 4K rw--- [ anon ]
>> 00007ffa42c69000 88K r-x-- libnet.so.1.7.0
>> 00007ffa42c7f000 2044K ----- libnet.so.1.7.0
>> 00007ffa42e7e000 4K r---- libnet.so.1.7.0
>> 00007ffa42e7f000 4K rw--- libnet.so.1.7.0
>> 00007ffa42e80000 8K rw--- [ anon ]
>> 00007ffa42e82000 132K r-x-- libafsocket.so
>> 00007ffa42ea3000 2048K ----- libafsocket.so
>> 00007ffa430a3000 4K r---- libafsocket.so
>> 00007ffa430a4000 8K rw--- libafsocket.so
>> 00007ffa430a6000 4K rw--- [ anon ]
>> 00007ffa430a7000 108K r-x-- libaffile.so
>> 00007ffa430c2000 2044K ----- libaffile.so
>> 00007ffa432c1000 4K r---- libaffile.so
>> 00007ffa432c2000 4K rw--- libaffile.so
>> 00007ffa432c3000 4K rw--- [ anon ]
>> 00007ffa432c4000 8K r-x-- libconfgen.so
>> 00007ffa432c6000 2044K ----- libconfgen.so
>> 00007ffa434c5000 4K r---- libconfgen.so
>> 00007ffa434c6000 4K rw--- libconfgen.so
>> 00007ffa434c7000 36K r-x-- libbasicfuncs.so
>> 00007ffa434d0000 2048K ----- libbasicfuncs.so
>> 00007ffa436d0000 4K r---- libbasicfuncs.so
>> 00007ffa436d1000 8K rw--- libbasicfuncs.so
>> 00007ffa436d3000 32K r-x-- libappmodel.so
>> 00007ffa436db000 2048K ----- libappmodel.so
>> 00007ffa438db000 4K r---- libappmodel.so
>> 00007ffa438dc000 4K rw--- libappmodel.so
>> 00007ffa438dd000 40K r-x-- libjson-c.so.2.0.0
>> 00007ffa438e7000 2044K ----- libjson-c.so.2.0.0
>> 00007ffa43ae6000 4K r---- libjson-c.so.2.0.0
>> 00007ffa43ae7000 4K rw--- libjson-c.so.2.0.0
>> 00007ffa43ae8000 40K r-x-- libjson-plugin.so
>> 00007ffa43af2000 2048K ----- libjson-plugin.so
>> 00007ffa43cf2000 4K r---- libjson-plugin.so
>> 00007ffa43cf3000 4K rw--- libjson-plugin.so
>> 00007ffa43cf4000 44K r-x-- libnss_nis-2.23.so
>> 00007ffa43cff000 2044K ----- libnss_nis-2.23.so
>> 00007ffa43efe000 4K r---- libnss_nis-2.23.so
>> 00007ffa43eff000 4K rw--- libnss_nis-2.23.so
>> 00007ffa43f00000 88K r-x-- libnsl-2.23.so
>> 00007ffa43f16000 2044K ----- libnsl-2.23.so
>> 00007ffa44115000 4K r---- libnsl-2.23.so
>> 00007ffa44116000 4K rw--- libnsl-2.23.so
>> 00007ffa44117000 8K rw--- [ anon ]
>> 00007ffa44119000 32K r-x-- libnss_compat-2.23.so
>> 00007ffa44121000 2044K ----- libnss_compat-2.23.so
>> 00007ffa44320000 4K r---- libnss_compat-2.23.so
>> 00007ffa44321000 4K rw--- libnss_compat-2.23.so
>> 00007ffa44322000 44K r-x-- libnss_files-2.23.so
>> 00007ffa4432d000 2044K ----- libnss_files-2.23.so
>> 00007ffa4452c000 4K r---- libnss_files-2.23.so
>> 00007ffa4452d000 4K rw--- libnss_files-2.23.so
>> 00007ffa4452e000 24K rw--- [ anon ]
>> 00007ffa44534000 72K r-x-- libgpg-error.so.0.17.0
>> 00007ffa44546000 2048K ----- libgpg-error.so.0.17.0
>> 00007ffa44746000 4K r---- libgpg-error.so.0.17.0
>> 00007ffa44747000 4K rw--- libgpg-error.so.0.17.0
>> 00007ffa44748000 864K r-x-- libgcrypt.so.20.0.5
>> 00007ffa44820000 2044K ----- libgcrypt.so.20.0.5
>> 00007ffa44a1f000 4K r---- libgcrypt.so.20.0.5
>> 00007ffa44a20000 32K rw--- libgcrypt.so.20.0.5
>> 00007ffa44a28000 4K rw--- [ anon ]
>> 00007ffa44a29000 132K r-x-- liblzma.so.5.0.0
>> 00007ffa44a4a000 2044K ----- liblzma.so.5.0.0
>> 00007ffa44c49000 4K r---- liblzma.so.5.0.0
>> 00007ffa44c4a000 4K rw--- liblzma.so.5.0.0
>> 00007ffa44c4b000 124K r-x-- libselinux.so.1
>> 00007ffa44c6a000 2044K ----- libselinux.so.1
>> 00007ffa44e69000 4K r---- libselinux.so.1
>> 00007ffa44e6a000 4K rw--- libselinux.so.1
>> 00007ffa44e6b000 8K rw--- [ anon ]
>> 00007ffa44e6d000 2152K r-x-- libcrypto.so.1.0.0
>> 00007ffa45087000 2044K ----- libcrypto.so.1.0.0
>> 00007ffa45286000 112K r---- libcrypto.so.1.0.0
>> 00007ffa452a2000 48K rw--- libcrypto.so.1.0.0
>> 00007ffa452ae000 12K rw--- [ anon ]
>> 00007ffa452b1000 376K r-x-- libssl.so.1.0.0
>> 00007ffa4530f000 2048K ----- libssl.so.1.0.0
>> 00007ffa4550f000 16K r---- libssl.so.1.0.0
>> 00007ffa45513000 28K rw--- libssl.so.1.0.0
>> 00007ffa4551a000 12K r-x-- libsecret-storage.so.0.0.0
>> 00007ffa4551d000 2044K ----- libsecret-storage.so.0.0.0
>> 00007ffa4571c000 4K r---- libsecret-storage.so.0.0.0
>> 00007ffa4571d000 4K rw--- libsecret-storage.so.0.0.0
>> 00007ffa4571e000 12K r-x-- libdl-2.23.so
>> 00007ffa45721000 2044K ----- libdl-2.23.so
>> 00007ffa45920000 4K r---- libdl-2.23.so
>> 00007ffa45921000 4K rw--- libdl-2.23.so
>> 00007ffa45922000 440K r-x-- libpcre.so.3.13.2
>> 00007ffa45990000 2048K ----- libpcre.so.3.13.2
>> 00007ffa45b90000 4K r---- libpcre.so.3.13.2
>> 00007ffa45b91000 4K rw--- libpcre.so.3.13.2
>> 00007ffa45b92000 16K r-x-- libcap.so.2.24
>> 00007ffa45b96000 2048K ----- libcap.so.2.24
>> 00007ffa45d96000 4K r---- libcap.so.2.24
>> 00007ffa45d97000 4K rw--- libcap.so.2.24
>> 00007ffa45d98000 12K r-x-- libevtlog-3.17.so.0.0.0
>> 00007ffa45d9b000 2048K ----- libevtlog-3.17.so.0.0.0
>> 00007ffa45f9b000 4K r---- libevtlog-3.17.so.0.0.0
>> 00007ffa45f9c000 4K rw--- libevtlog-3.17.so.0.0.0
>> 00007ffa45f9d000 4K r-x-- libgthread-2.0.so.0.4800.2
>> 00007ffa45f9e000 2044K ----- libgthread-2.0.so.0.4800.2
>> 00007ffa4619d000 4K r---- libgthread-2.0.so.0.4800.2
>> 00007ffa4619e000 4K rw--- libgthread-2.0.so.0.4800.2
>> 00007ffa4619f000 12K r-x-- libgmodule-2.0.so.0.4800.2
>> 00007ffa461a2000 2044K ----- libgmodule-2.0.so.0.4800.2
>> 00007ffa463a1000 4K r---- libgmodule-2.0.so.0.4800.2
>> 00007ffa463a2000 4K rw--- libgmodule-2.0.so.0.4800.2
>> 00007ffa463a3000 28K r-x-- librt-2.23.so
>> 00007ffa463aa000 2044K ----- librt-2.23.so
>> 00007ffa465a9000 4K r---- librt-2.23.so
>> 00007ffa465aa000 4K rw--- librt-2.23.so
>> 00007ffa465ab000 1792K r-x-- libc-2.23.so
>> 00007ffa4676b000 2044K ----- libc-2.23.so
>> 00007ffa4696a000 16K r---- libc-2.23.so
>> 00007ffa4696e000 8K rw--- libc-2.23.so
>> 00007ffa46970000 16K rw--- [ anon ]
>> 00007ffa46974000 96K r-x-- libpthread-2.23.so
>> 00007ffa4698c000 2044K ----- libpthread-2.23.so
>> 00007ffa46b8b000 4K r---- libpthread-2.23.so
>> 00007ffa46b8c000 4K rw--- libpthread-2.23.so
>> 00007ffa46b8d000 16K rw--- [ anon ]
>> 00007ffa46b91000 1084K r-x-- libglib-2.0.so.0.4800.2
>> 00007ffa46ca0000 2044K ----- libglib-2.0.so.0.4800.2
>> 00007ffa46e9f000 4K r---- libglib-2.0.so.0.4800.2
>> 00007ffa46ea0000 4K rw--- libglib-2.0.so.0.4800.2
>> 00007ffa46ea1000 4K rw--- [ anon ]
>> 00007ffa46ea2000 768K r-x-- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa46f62000 2044K ----- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa47161000 16K r---- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa47165000 24K rw--- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa4716b000 8K rw--- [ anon ]
>> 00007ffa4716d000 152K r-x-- ld-2.23.so
>> 00007ffa471ba000 256K rw--- [ anon ]
>> 00007ffa472a2000 256K rw--- [ anon ]
>> 00007ffa472e2000 64K rwx-- [ anon ]
>> 00007ffa472f2000 28K r--s- gconv-modules.cache
>> 00007ffa472f9000 36K rw--- [ anon ]
>> 00007ffa47302000 512K r-x-- libsystemd.so.0.14.0
>> 00007ffa47382000 12K r---- libsystemd.so.0.14.0
>> 00007ffa47385000 4K rw--- libsystemd.so.0.14.0
>> 00007ffa47386000 20K rw--- [ anon ]
>> 00007ffa47390000 8K rw--- [ anon ]
>> 00007ffa47392000 4K r---- ld-2.23.so
>> 00007ffa47393000 4K rw--- ld-2.23.so
>> 00007ffa47394000 4K rw--- [ anon ]
>> 00007ffe81489000 132K rw--- [ stack ]
>> 00007ffe81564000 8K r-x-- [ anon ]
>> ffffffffff600000 4K r-x-- [ anon ]
>> total 7226528K
>>
>>
>> Configuration:
>>
>> @version: 3.16.1
>> @module mod-java
>> @include "/etc/syslog-ng/scl.conf"
>>
>> options {
>> chain_hostnames(off);
>> use-dns(no);
>> use-fqdn(no);
>> owner("syslog-ng");
>> group("stratio");
>> perm(0640);
>> stats_freq(0);
>> bad-hostname("^gconfd$");
>> flush-lines(100);
>> log-fifo-size(345000);
>> };
>>
>> ########################
>> # Filters
>> ########################
>> # Here's come the filter options. With this rules, we can set which
>> # message go where.
>>
>> filter f_dbg { level(debug); };
>> filter f_info { level(info); };
>> filter f_notice { level(notice); };
>> filter f_warn { level(warn); };
>> filter f_err { level(err); };
>> filter f_crit { level(crit .. emerg); };
>>
>> filter f_debug { level(debug) and not facility(auth, authpriv, news,
>> mail); };
>> filter f_error { level(err .. emerg) ; };
>> filter f_messages { level(info,notice,warn) and
>> not facility(auth,authpriv,cron,daemon,mail,news); };
>>
>> filter f_auth { facility(auth, authpriv) and not filter(f_debug); };
>> filter f_cron { facility(cron) and not filter(f_debug); };
>> filter f_daemon { facility(daemon) and not filter(f_debug); };
>> filter f_kern { facility(kern) and not filter(f_debug); };
>> filter f_lpr { facility(lpr) and not filter(f_debug); };
>> filter f_local { facility(local0, local1, local3, local4, local5,
>> local6, local7) and not filter(f_debug); };
>> filter f_mail { facility(mail) and not filter(f_debug); };
>> filter f_news { facility(news) and not filter(f_debug); };
>> filter f_syslog3 { not facility(auth, authpriv, mail) and not
>> filter(f_debug); };
>> filter f_user { facility(user) and not filter(f_debug); };
>>
>> filter f_text_only {
>> not match(
>> ".*[\\x80-\\xFF].*"
>> value("MESSAGE")
>> );
>> };
>>
>> ###
>> # Sources
>>
>> source s_stdout_files { wildcard-file(
>> base-dir("/var/lib/slaves/")
>> filename-pattern("stdout")
>> max-files(3400)
>> follow-freq(0.5)
>> monitor-method(poll)
>> log-fetch-limit(100)
>> log-iw-size(345000)
>> flags(no-parse)
>> recursive(yes)
>> multi-line-mode(indented)
>> ); };
>>
>> source s_stderr_files { wildcard-file(
>> base-dir("/var/lib/slaves/")
>> filename-pattern("stderr")
>> max-files(3400)
>> follow-freq(0.5)
>> monitor-method(poll)
>> log-fetch-limit(100)
>> log-iw-size(345000)
>> flags(no-parse)
>> recursive(yes)
>> multi-line-mode(indented)
>> ); };
>>
>>
>> Thanks.
>>
>>
>> 2018-09-06 12:59 GMT+02:00 Nemes, Antal <antal.nemes at oneidentity.com>:
>>
>>> Hi,
>>>
>>> The same free policy cannot work with _handle_file_deleted, because we
>>> cannot free the readers immediately. There is a race between file delete
>>> and syslog-ng reading the data. It can happen that one deletes a file
>>> before syslog-ng reads all its contents, which would cause message loss. In
>>> that case we just signal the deletion event to the reader
>>> (log_pipe_notify(&reader->super, NC_FILE_DELETED, NULL); and when the
>>> reader reads EOF at a later time, that's when the reader object is freed.
>>>
>>> Can you send the output of pmap <pid of syslog-ng> with the leak? That
>>> may give us a clue where to look with the leak.
>>>
>>> Also can you send your configuration, or at least the part about the
>>> wildcard filesources? The reproduction might depend on max_files or
>>> monitor_method parameter, or maybe other.
>>>
>>> Thanks,
>>> Antal
>>>
>>>
>>> On Thu, Sep 6, 2018 at 12:00 PM Jose Angel Santiago <
>>> jasantiago at stratio.com> wrote:
>>>
>>>> Reviewing your commit I've found out that the method
>>>> *_handle_file_deleted* seems not to free memory when a monitored file
>>>> has been deleted, according to the changes you made in
>>>> *_handler_handler_directory_deleted.* Could it be the cause of the
>>>> memory leak?
>>>>
>>>> Thanks.
>>>>
>>>> 2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago at stratio.com>
>>>> :
>>>>
>>>>> Hi,
>>>>>
>>>>> I was testing your fix in several environments before replying,
>>>>> unfortunately syslog-ng stills shows a huge RAM consumption in our use of
>>>>> case.
>>>>>
>>>>> We have two wildcard-file sources defined in our syslog-ng conf file,
>>>>> both of them have the same base-dir reference, but one source has a
>>>>> filename-pattern and the other source has a different one (I think we can't
>>>>> use the same source and several filename-patterns because of the lack of
>>>>> regular expression support). We execute the deletion of obsolete files and
>>>>> folder under base-dir once every 2 days, and we create a lot of new folders
>>>>> and files constantly while using our platform, but the growth of these log
>>>>> files (the ones affected by the filename-pattern) has been pretty low in
>>>>> this test, about 10-20 msgs per second total.
>>>>>
>>>>> This is some data of the fix test:
>>>>>
>>>>> Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the
>>>>> filename-pattern and 20000 folders under base-dir (not a single folder were
>>>>> deleted yet so this is the expected behaviour of syslog-ng, and we find it
>>>>> extremely unaffordable)
>>>>>
>>>>> One hour later, right before the cleaning process was executed,
>>>>> syslog-ng was consuming 35G of RAM, and it didn't decrease after the
>>>>> cleaning, not even a little, with just 380 files affected by the
>>>>> filename-pattern and 2200 folders under base-dir.
>>>>>
>>>>> There is no message in the syslog-agent queue and all the messages are
>>>>> received by the relay (which ingest messages in an elasticsearch)
>>>>>
>>>>> For the record, we do noticed a small RAM usage decrease in a previous
>>>>> test in a lab environment when we deleted every folder & file under
>>>>> base-dir (from 870M to 550M).
>>>>>
>>>>> It seems syslog-ng wildcard-source is not optimized when it has too
>>>>> handle thousands of folders and filename-patterns under base-dir. Have you
>>>>> ever test the wildcard-source with a use of case similar to ours?
>>>>>
>>>>> Can I send you any more information which may help to find out how to
>>>>> optimize it? I wish I can help with the code but I have no idea of C
>>>>> language :-(
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy at oneidentity.com>:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Thanks for the thorough investigation!
>>>>>> We've checked it and found a memory leak in directory monitoring.
>>>>>>
>>>>>> I've pushed the fix to my fork and created a merge request about it:
>>>>>> https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak
>>>>>> https://github.com/balabit/syslog-ng/pull/2261
>>>>>> Can you verify that this fixes your problem, please?
>>>>>>
>>>>>> > log-fifo-size value is 345000, so I assume it can't be a buffer
>>>>>> situation since 345000 messages can't occupy 40-50 GB of memory.
>>>>>> Well our default log-msg-size is 64kB so if all messages have a size
>>>>>> of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
>>>>>>
>>>>>> Best Regards,
>>>>>> Gabor
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago <
>>>>>> jasantiago at stratio.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call
>>>>>>> it "syslog-agent), which sends log messages to another syslog-ng acting as
>>>>>>> a relay.
>>>>>>>
>>>>>>> I've noticed that syslog-agent instances RAM consumption keeps
>>>>>>> increasing until they leave no free memory in the cluster (each server has
>>>>>>> 64G of RAM). In my use case, new folders & files are created constantly
>>>>>>> under base-dir folder, but every 2 days obsolete folders & files are
>>>>>>> deleted. I assumed that syslog-ng would free some RAM every time those
>>>>>>> folders & files are deleted, but it doesn't, not even if I run a
>>>>>>> syslog-ng-ctl reload operation.
>>>>>>>
>>>>>>> log-fifo-size value is 345000, so I assume it can't be a buffer
>>>>>>> situation since 345000 messages can't occupy 40-50 GB of memory.
>>>>>>>
>>>>>>> I've performed the following test to reproduce the situation in
>>>>>>> small scale:
>>>>>>>
>>>>>>> - Launch a syslog-agent with a wildcard-file source reading from
>>>>>>> "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M.
>>>>>>> - Run a simple script to create complex folder hierarchy under
>>>>>>> /tmp/test and some files with 5000 log messages to read from.
>>>>>>> - Wait until syslog-agent RAM usage gets 1GB
>>>>>>> - Stop script execution and wait until syslog-agent has send all
>>>>>>> logs to relay.
>>>>>>> - Delete everything under /tmp/test and execute syslog-ng-ctl reload
>>>>>>> operation
>>>>>>> - 24 hours after, syslog-agent RAM usage still is 1GB
>>>>>>>
>>>>>>> I've used heaptrack tool as a try to find a memory leak in
>>>>>>> syslog-ng, you can see in the attached image that iv_list_empty function in
>>>>>>> iv_list.h file is where most of the RAM usage is.
>>>>>>>
>>>>>>> How do I get syslog-ng to free RAM? Or is it a memory leak?
>>>>>>>
>>>>>>> Thanks in advance.
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> | Jose Angel Santiago
>>>>>>>
>>>>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>>>>
>>>>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>>>>
>>>>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>>>>
>>>>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>>>>> <https://twitter.com/stratiobd>
>>>>>>> <https://www.linkedin.com/company/stratiobd>
>>>>>>> <https://www.youtube.com/c/StratioBD>
>>>>>>> ____________________________________________________________
>>>>>>> __________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>>>>> product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ____________________________________________________________
>>>>>> __________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>>>> product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> | Jose Angel Santiago
>>>>>
>>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>>
>>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>>
>>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>>
>>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>>> <https://twitter.com/stratiobd>
>>>>> <https://www.linkedin.com/company/stratiobd>
>>>>> <https://www.youtube.com/c/StratioBD>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> | Jose Angel Santiago
>>>>
>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>
>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>
>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>
>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>> <https://twitter.com/stratiobd>
>>>> <https://www.linkedin.com/company/stratiobd>
>>>> <https://www.youtube.com/c/StratioBD>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>> product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?
>>> product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> --
>>
>> | Jose Angel Santiago
>>
>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>
>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>
>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>
>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>> <https://twitter.com/stratiobd>
>> <https://www.linkedin.com/company/stratiobd>
>> <https://www.youtube.com/c/StratioBD>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com
<https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd>
<https://www.youtube.com/c/StratioBD>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180910/841c6c20/attachment-0001.html>
More information about the syslog-ng
mailing list