[syslog-ng] syslog-ng never frees RAM
Nemes, Antal
antal.nemes at oneidentity.com
Thu Sep 6 14:45:21 UTC 2018
Hi,
We found another leak related to monitor-method(poll). When the tracked
directory is deleted, glib allocates a GError structure for "Error opening
directory \'...\': No such file or directory", that was not freed by
syslog-ng. This can cause considerable memory leak when high number of
directories tracked and deleted. We created a pull request to fix it: here
you can track the progress: https://github.com/balabit/syslog-ng/pull/2267
Br,
Antal
On Thu, Sep 6, 2018 at 1:15 PM Jose Angel Santiago <jasantiago at stratio.com>
wrote:
> Hi,
>
> I had to restart syslog-ng process because it was affecting other services
> in the same host.
>
> I've checked another host with was running with the fix too, it has 23000
> files monitored and 38700 folders under base-dir, and now it's starting to
> eat the RAM, the cleaning process will be executed in about an hour (I'm
> sorry I can't force to do it before)
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 96 root 20 0 *5.609g 4.274g* 0.006g R *101.3* *6.8 * 1263:46
> syslog-ng
>
> Trace of pmap:
>
> 96: /syslog-ng/exec/sbin/syslog-ng -F --no-caps -f
> /etc/syslog-ng/syslog-ng.conf -p /var/syslog-ng/syslog-ng.pid -R
> /var/syslog-ng/syslog-ng.persist -c /var/syslog-ng/syslog-ng.ctl
> 0000000000400000 12K r-x-- syslog-ng
> 0000000000602000 4K r---- syslog-ng
> 0000000000603000 4K rw--- syslog-ng
> 00000000015bc000 5689360K rw--- [ anon ]
> 00007ff9c8000000 19588K rw--- [ anon ]
> 00007ff9c9321000 45948K ----- [ anon ]
> 00007ff9cc000000 4288K rw--- [ anon ]
> 00007ff9cc430000 61248K ----- [ anon ]
> 00007ff9d0000000 26844K rw--- [ anon ]
> 00007ff9d1a37000 38692K ----- [ anon ]
> 00007ff9d4000000 10280K rw--- [ anon ]
> 00007ff9d4a0a000 55256K ----- [ anon ]
> 00007ff9dc000000 64604K rw--- [ anon ]
> 00007ff9dff17000 932K ----- [ anon ]
> 00007ff9e0000000 65508K rw--- [ anon ]
> 00007ff9e3ff9000 28K ----- [ anon ]
> 00007ff9e8000000 4480K rw--- [ anon ]
> 00007ff9e8460000 61056K ----- [ anon ]
> 00007ff9f0000000 65488K rw--- [ anon ]
> 00007ff9f3ff4000 48K ----- [ anon ]
> 00007ff9f8000000 14716K rw--- [ anon ]
> 00007ff9f8e5f000 50820K ----- [ anon ]
> 00007ffa00000000 18252K rw--- [ anon ]
> 00007ffa011d3000 47284K ----- [ anon ]
> 00007ffa08000000 65536K rw--- [ anon ]
> 00007ffa0c000000 30460K rw--- [ anon ]
> 00007ffa0ddbf000 35076K ----- [ anon ]
> 00007ffa10000000 65484K rw--- [ anon ]
> 00007ffa13ff3000 52K ----- [ anon ]
> 00007ffa167fd000 4K ----- [ anon ]
> 00007ffa167fe000 8192K rw--- [ anon ]
> 00007ffa18000000 7920K rw--- [ anon ]
> 00007ffa187bc000 57616K ----- [ anon ]
> 00007ffa1cffa000 4K ----- [ anon ]
> 00007ffa1cffb000 8192K rw--- [ anon ]
> 00007ffa1f7ff000 4K ----- [ anon ]
> 00007ffa1f800000 8192K rw--- [ anon ]
> 00007ffa20000000 65536K rw--- [ anon ]
> 00007ffa24000000 14764K rw--- [ anon ]
> 00007ffa24e6b000 50772K ----- [ anon ]
> 00007ffa28000000 24348K rw--- [ anon ]
> 00007ffa297c7000 41188K ----- [ anon ]
> 00007ffa2c000000 40916K rw--- [ anon ]
> 00007ffa2e7f5000 24620K ----- [ anon ]
> 00007ffa30000000 17684K rw--- [ anon ]
> 00007ffa31145000 47852K ----- [ anon ]
> 00007ffa34000000 10988K rw--- [ anon ]
> 00007ffa34abb000 54548K ----- [ anon ]
> 00007ffa38000000 20464K rw--- [ anon ]
> 00007ffa393fc000 45072K ----- [ anon ]
> 00007ffa3c316000 4K ----- [ anon ]
> 00007ffa3c317000 8192K rw--- [ anon ]
> 00007ffa3ea78000 4K ----- [ anon ]
> 00007ffa3ea79000 8192K rw--- [ anon ]
> 00007ffa3fbee000 7360K rw-s- syslog-ng.persist
> 00007ffa4031e000 92K r-x-- libresolv-2.23.so
> 00007ffa40335000 2048K ----- libresolv-2.23.so
> 00007ffa40535000 4K r---- libresolv-2.23.so
> 00007ffa40536000 4K rw--- libresolv-2.23.so
> 00007ffa40537000 8K rw--- [ anon ]
> 00007ffa40539000 20K r-x-- libnss_dns-2.23.so
> 00007ffa4053e000 2048K ----- libnss_dns-2.23.so
> 00007ffa4073e000 4K r---- libnss_dns-2.23.so
> 00007ffa4073f000 4K rw--- libnss_dns-2.23.so
> 00007ffa40740000 3328K rw--- [ anon ]
> 00007ffa40ac0000 256K rw--- [ anon ]
> 00007ffa40b40000 512K rw--- [ anon ]
> 00007ffa40c00000 256K rw--- [ anon ]
> 00007ffa40c6c000 256K rw--- [ anon ]
> 00007ffa40d6c000 44K r-x-- libsyslogformat.so
> 00007ffa40d77000 2044K ----- libsyslogformat.so
> 00007ffa40f76000 4K r---- libsyslogformat.so
> 00007ffa40f77000 4K rw--- libsyslogformat.so
> 00007ffa40f78000 4612K rw--- [ anon ]
> 00007ffa413f9000 24K r-x-- _hashlib.x86_64-linux-gnu.so
> 00007ffa413ff000 2044K ----- _hashlib.x86_64-linux-gnu.so
> 00007ffa415fe000 4K r---- _hashlib.x86_64-linux-gnu.so
> 00007ffa415ff000 4K rw--- _hashlib.x86_64-linux-gnu.so
> 00007ffa41600000 256K rw--- [ anon ]
> 00007ffa41640000 48K r-x-- _json.x86_64-linux-gnu.so
> 00007ffa4164c000 2044K ----- _json.x86_64-linux-gnu.so
> 00007ffa4184b000 4K r---- _json.x86_64-linux-gnu.so
> 00007ffa4184c000 4K rw--- _json.x86_64-linux-gnu.so
> 00007ffa4184d000 84K r-x-- _ssl.x86_64-linux-gnu.so
> 00007ffa41862000 2044K ----- _ssl.x86_64-linux-gnu.so
> 00007ffa41a61000 4K r---- _ssl.x86_64-linux-gnu.so
> 00007ffa41a62000 16K rw--- _ssl.x86_64-linux-gnu.so
> 00007ffa41a66000 768K rw--- [ anon ]
> 00007ffa41b57000 256K rw--- [ anon ]
> 00007ffa41b97000 1056K r-x-- libm-2.23.so
> 00007ffa41c9f000 2044K ----- libm-2.23.so
> 00007ffa41e9e000 4K r---- libm-2.23.so
> 00007ffa41e9f000 4K rw--- libm-2.23.so
> 00007ffa41ea0000 8K r-x-- libutil-2.23.so
> 00007ffa41ea2000 2044K ----- libutil-2.23.so
> 00007ffa420a1000 4K r---- libutil-2.23.so
> 00007ffa420a2000 4K rw--- libutil-2.23.so
> 00007ffa420a3000 100K r-x-- libz.so.1.2.8
> 00007ffa420bc000 2044K ----- libz.so.1.2.8
> 00007ffa422bb000 4K r---- libz.so.1.2.8
> 00007ffa422bc000 4K rw--- libz.so.1.2.8
> 00007ffa422bd000 3016K r-x-- libpython2.7.so.1.0
> 00007ffa425af000 2048K ----- libpython2.7.so.1.0
> 00007ffa427af000 8K r---- libpython2.7.so.1.0
> 00007ffa427b1000 476K rw--- libpython2.7.so.1.0
> 00007ffa42828000 140K rw--- [ anon ]
> 00007ffa4284b000 72K r-x-- libmod-python.so
> 00007ffa4285d000 2044K ----- libmod-python.so
> 00007ffa42a5c000 4K r---- libmod-python.so
> 00007ffa42a5d000 4K rw--- libmod-python.so
> 00007ffa42a5e000 4K rw--- [ anon ]
> 00007ffa42a5f000 32K r-x-- libwrap.so.0.7.6
> 00007ffa42a67000 2044K ----- libwrap.so.0.7.6
> 00007ffa42c66000 4K r---- libwrap.so.0.7.6
> 00007ffa42c67000 4K rw--- libwrap.so.0.7.6
> 00007ffa42c68000 4K rw--- [ anon ]
> 00007ffa42c69000 88K r-x-- libnet.so.1.7.0
> 00007ffa42c7f000 2044K ----- libnet.so.1.7.0
> 00007ffa42e7e000 4K r---- libnet.so.1.7.0
> 00007ffa42e7f000 4K rw--- libnet.so.1.7.0
> 00007ffa42e80000 8K rw--- [ anon ]
> 00007ffa42e82000 132K r-x-- libafsocket.so
> 00007ffa42ea3000 2048K ----- libafsocket.so
> 00007ffa430a3000 4K r---- libafsocket.so
> 00007ffa430a4000 8K rw--- libafsocket.so
> 00007ffa430a6000 4K rw--- [ anon ]
> 00007ffa430a7000 108K r-x-- libaffile.so
> 00007ffa430c2000 2044K ----- libaffile.so
> 00007ffa432c1000 4K r---- libaffile.so
> 00007ffa432c2000 4K rw--- libaffile.so
> 00007ffa432c3000 4K rw--- [ anon ]
> 00007ffa432c4000 8K r-x-- libconfgen.so
> 00007ffa432c6000 2044K ----- libconfgen.so
> 00007ffa434c5000 4K r---- libconfgen.so
> 00007ffa434c6000 4K rw--- libconfgen.so
> 00007ffa434c7000 36K r-x-- libbasicfuncs.so
> 00007ffa434d0000 2048K ----- libbasicfuncs.so
> 00007ffa436d0000 4K r---- libbasicfuncs.so
> 00007ffa436d1000 8K rw--- libbasicfuncs.so
> 00007ffa436d3000 32K r-x-- libappmodel.so
> 00007ffa436db000 2048K ----- libappmodel.so
> 00007ffa438db000 4K r---- libappmodel.so
> 00007ffa438dc000 4K rw--- libappmodel.so
> 00007ffa438dd000 40K r-x-- libjson-c.so.2.0.0
> 00007ffa438e7000 2044K ----- libjson-c.so.2.0.0
> 00007ffa43ae6000 4K r---- libjson-c.so.2.0.0
> 00007ffa43ae7000 4K rw--- libjson-c.so.2.0.0
> 00007ffa43ae8000 40K r-x-- libjson-plugin.so
> 00007ffa43af2000 2048K ----- libjson-plugin.so
> 00007ffa43cf2000 4K r---- libjson-plugin.so
> 00007ffa43cf3000 4K rw--- libjson-plugin.so
> 00007ffa43cf4000 44K r-x-- libnss_nis-2.23.so
> 00007ffa43cff000 2044K ----- libnss_nis-2.23.so
> 00007ffa43efe000 4K r---- libnss_nis-2.23.so
> 00007ffa43eff000 4K rw--- libnss_nis-2.23.so
> 00007ffa43f00000 88K r-x-- libnsl-2.23.so
> 00007ffa43f16000 2044K ----- libnsl-2.23.so
> 00007ffa44115000 4K r---- libnsl-2.23.so
> 00007ffa44116000 4K rw--- libnsl-2.23.so
> 00007ffa44117000 8K rw--- [ anon ]
> 00007ffa44119000 32K r-x-- libnss_compat-2.23.so
> 00007ffa44121000 2044K ----- libnss_compat-2.23.so
> 00007ffa44320000 4K r---- libnss_compat-2.23.so
> 00007ffa44321000 4K rw--- libnss_compat-2.23.so
> 00007ffa44322000 44K r-x-- libnss_files-2.23.so
> 00007ffa4432d000 2044K ----- libnss_files-2.23.so
> 00007ffa4452c000 4K r---- libnss_files-2.23.so
> 00007ffa4452d000 4K rw--- libnss_files-2.23.so
> 00007ffa4452e000 24K rw--- [ anon ]
> 00007ffa44534000 72K r-x-- libgpg-error.so.0.17.0
> 00007ffa44546000 2048K ----- libgpg-error.so.0.17.0
> 00007ffa44746000 4K r---- libgpg-error.so.0.17.0
> 00007ffa44747000 4K rw--- libgpg-error.so.0.17.0
> 00007ffa44748000 864K r-x-- libgcrypt.so.20.0.5
> 00007ffa44820000 2044K ----- libgcrypt.so.20.0.5
> 00007ffa44a1f000 4K r---- libgcrypt.so.20.0.5
> 00007ffa44a20000 32K rw--- libgcrypt.so.20.0.5
> 00007ffa44a28000 4K rw--- [ anon ]
> 00007ffa44a29000 132K r-x-- liblzma.so.5.0.0
> 00007ffa44a4a000 2044K ----- liblzma.so.5.0.0
> 00007ffa44c49000 4K r---- liblzma.so.5.0.0
> 00007ffa44c4a000 4K rw--- liblzma.so.5.0.0
> 00007ffa44c4b000 124K r-x-- libselinux.so.1
> 00007ffa44c6a000 2044K ----- libselinux.so.1
> 00007ffa44e69000 4K r---- libselinux.so.1
> 00007ffa44e6a000 4K rw--- libselinux.so.1
> 00007ffa44e6b000 8K rw--- [ anon ]
> 00007ffa44e6d000 2152K r-x-- libcrypto.so.1.0.0
> 00007ffa45087000 2044K ----- libcrypto.so.1.0.0
> 00007ffa45286000 112K r---- libcrypto.so.1.0.0
> 00007ffa452a2000 48K rw--- libcrypto.so.1.0.0
> 00007ffa452ae000 12K rw--- [ anon ]
> 00007ffa452b1000 376K r-x-- libssl.so.1.0.0
> 00007ffa4530f000 2048K ----- libssl.so.1.0.0
> 00007ffa4550f000 16K r---- libssl.so.1.0.0
> 00007ffa45513000 28K rw--- libssl.so.1.0.0
> 00007ffa4551a000 12K r-x-- libsecret-storage.so.0.0.0
> 00007ffa4551d000 2044K ----- libsecret-storage.so.0.0.0
> 00007ffa4571c000 4K r---- libsecret-storage.so.0.0.0
> 00007ffa4571d000 4K rw--- libsecret-storage.so.0.0.0
> 00007ffa4571e000 12K r-x-- libdl-2.23.so
> 00007ffa45721000 2044K ----- libdl-2.23.so
> 00007ffa45920000 4K r---- libdl-2.23.so
> 00007ffa45921000 4K rw--- libdl-2.23.so
> 00007ffa45922000 440K r-x-- libpcre.so.3.13.2
> 00007ffa45990000 2048K ----- libpcre.so.3.13.2
> 00007ffa45b90000 4K r---- libpcre.so.3.13.2
> 00007ffa45b91000 4K rw--- libpcre.so.3.13.2
> 00007ffa45b92000 16K r-x-- libcap.so.2.24
> 00007ffa45b96000 2048K ----- libcap.so.2.24
> 00007ffa45d96000 4K r---- libcap.so.2.24
> 00007ffa45d97000 4K rw--- libcap.so.2.24
> 00007ffa45d98000 12K r-x-- libevtlog-3.17.so.0.0.0
> 00007ffa45d9b000 2048K ----- libevtlog-3.17.so.0.0.0
> 00007ffa45f9b000 4K r---- libevtlog-3.17.so.0.0.0
> 00007ffa45f9c000 4K rw--- libevtlog-3.17.so.0.0.0
> 00007ffa45f9d000 4K r-x-- libgthread-2.0.so.0.4800.2
> 00007ffa45f9e000 2044K ----- libgthread-2.0.so.0.4800.2
> 00007ffa4619d000 4K r---- libgthread-2.0.so.0.4800.2
> 00007ffa4619e000 4K rw--- libgthread-2.0.so.0.4800.2
> 00007ffa4619f000 12K r-x-- libgmodule-2.0.so.0.4800.2
> 00007ffa461a2000 2044K ----- libgmodule-2.0.so.0.4800.2
> 00007ffa463a1000 4K r---- libgmodule-2.0.so.0.4800.2
> 00007ffa463a2000 4K rw--- libgmodule-2.0.so.0.4800.2
> 00007ffa463a3000 28K r-x-- librt-2.23.so
> 00007ffa463aa000 2044K ----- librt-2.23.so
> 00007ffa465a9000 4K r---- librt-2.23.so
> 00007ffa465aa000 4K rw--- librt-2.23.so
> 00007ffa465ab000 1792K r-x-- libc-2.23.so
> 00007ffa4676b000 2044K ----- libc-2.23.so
> 00007ffa4696a000 16K r---- libc-2.23.so
> 00007ffa4696e000 8K rw--- libc-2.23.so
> 00007ffa46970000 16K rw--- [ anon ]
> 00007ffa46974000 96K r-x-- libpthread-2.23.so
> 00007ffa4698c000 2044K ----- libpthread-2.23.so
> 00007ffa46b8b000 4K r---- libpthread-2.23.so
> 00007ffa46b8c000 4K rw--- libpthread-2.23.so
> 00007ffa46b8d000 16K rw--- [ anon ]
> 00007ffa46b91000 1084K r-x-- libglib-2.0.so.0.4800.2
> 00007ffa46ca0000 2044K ----- libglib-2.0.so.0.4800.2
> 00007ffa46e9f000 4K r---- libglib-2.0.so.0.4800.2
> 00007ffa46ea0000 4K rw--- libglib-2.0.so.0.4800.2
> 00007ffa46ea1000 4K rw--- [ anon ]
> 00007ffa46ea2000 768K r-x-- libsyslog-ng-3.17.so.0.0.0
> 00007ffa46f62000 2044K ----- libsyslog-ng-3.17.so.0.0.0
> 00007ffa47161000 16K r---- libsyslog-ng-3.17.so.0.0.0
> 00007ffa47165000 24K rw--- libsyslog-ng-3.17.so.0.0.0
> 00007ffa4716b000 8K rw--- [ anon ]
> 00007ffa4716d000 152K r-x-- ld-2.23.so
> 00007ffa471ba000 256K rw--- [ anon ]
> 00007ffa472a2000 256K rw--- [ anon ]
> 00007ffa472e2000 64K rwx-- [ anon ]
> 00007ffa472f2000 28K r--s- gconv-modules.cache
> 00007ffa472f9000 36K rw--- [ anon ]
> 00007ffa47302000 512K r-x-- libsystemd.so.0.14.0
> 00007ffa47382000 12K r---- libsystemd.so.0.14.0
> 00007ffa47385000 4K rw--- libsystemd.so.0.14.0
> 00007ffa47386000 20K rw--- [ anon ]
> 00007ffa47390000 8K rw--- [ anon ]
> 00007ffa47392000 4K r---- ld-2.23.so
> 00007ffa47393000 4K rw--- ld-2.23.so
> 00007ffa47394000 4K rw--- [ anon ]
> 00007ffe81489000 132K rw--- [ stack ]
> 00007ffe81564000 8K r-x-- [ anon ]
> ffffffffff600000 4K r-x-- [ anon ]
> total 7226528K
>
>
> Configuration:
>
> @version: 3.16.1
> @module mod-java
> @include "/etc/syslog-ng/scl.conf"
>
> options {
> chain_hostnames(off);
> use-dns(no);
> use-fqdn(no);
> owner("syslog-ng");
> group("stratio");
> perm(0640);
> stats_freq(0);
> bad-hostname("^gconfd$");
> flush-lines(100);
> log-fifo-size(345000);
> };
>
> ########################
> # Filters
> ########################
> # Here's come the filter options. With this rules, we can set which
> # message go where.
>
> filter f_dbg { level(debug); };
> filter f_info { level(info); };
> filter f_notice { level(notice); };
> filter f_warn { level(warn); };
> filter f_err { level(err); };
> filter f_crit { level(crit .. emerg); };
>
> filter f_debug { level(debug) and not facility(auth, authpriv, news,
> mail); };
> filter f_error { level(err .. emerg) ; };
> filter f_messages { level(info,notice,warn) and
> not facility(auth,authpriv,cron,daemon,mail,news); };
>
> filter f_auth { facility(auth, authpriv) and not filter(f_debug); };
> filter f_cron { facility(cron) and not filter(f_debug); };
> filter f_daemon { facility(daemon) and not filter(f_debug); };
> filter f_kern { facility(kern) and not filter(f_debug); };
> filter f_lpr { facility(lpr) and not filter(f_debug); };
> filter f_local { facility(local0, local1, local3, local4, local5,
> local6, local7) and not filter(f_debug); };
> filter f_mail { facility(mail) and not filter(f_debug); };
> filter f_news { facility(news) and not filter(f_debug); };
> filter f_syslog3 { not facility(auth, authpriv, mail) and not
> filter(f_debug); };
> filter f_user { facility(user) and not filter(f_debug); };
>
> filter f_text_only {
> not match(
> ".*[\\x80-\\xFF].*"
> value("MESSAGE")
> );
> };
>
> ###
> # Sources
>
> source s_stdout_files { wildcard-file(
> base-dir("/var/lib/slaves/")
> filename-pattern("stdout")
> max-files(3400)
> follow-freq(0.5)
> monitor-method(poll)
> log-fetch-limit(100)
> log-iw-size(345000)
> flags(no-parse)
> recursive(yes)
> multi-line-mode(indented)
> ); };
>
> source s_stderr_files { wildcard-file(
> base-dir("/var/lib/slaves/")
> filename-pattern("stderr")
> max-files(3400)
> follow-freq(0.5)
> monitor-method(poll)
> log-fetch-limit(100)
> log-iw-size(345000)
> flags(no-parse)
> recursive(yes)
> multi-line-mode(indented)
> ); };
>
>
> Thanks.
>
>
> 2018-09-06 12:59 GMT+02:00 Nemes, Antal <antal.nemes at oneidentity.com>:
>
>> Hi,
>>
>> The same free policy cannot work with _handle_file_deleted, because we
>> cannot free the readers immediately. There is a race between file delete
>> and syslog-ng reading the data. It can happen that one deletes a file
>> before syslog-ng reads all its contents, which would cause message loss. In
>> that case we just signal the deletion event to the reader
>> (log_pipe_notify(&reader->super, NC_FILE_DELETED, NULL); and when the
>> reader reads EOF at a later time, that's when the reader object is freed.
>>
>> Can you send the output of pmap <pid of syslog-ng> with the leak? That
>> may give us a clue where to look with the leak.
>>
>> Also can you send your configuration, or at least the part about the
>> wildcard filesources? The reproduction might depend on max_files or
>> monitor_method parameter, or maybe other.
>>
>> Thanks,
>> Antal
>>
>>
>> On Thu, Sep 6, 2018 at 12:00 PM Jose Angel Santiago <
>> jasantiago at stratio.com> wrote:
>>
>>> Reviewing your commit I've found out that the method
>>> *_handle_file_deleted* seems not to free memory when a monitored file
>>> has been deleted, according to the changes you made in
>>> *_handler_handler_directory_deleted.* Could it be the cause of the
>>> memory leak?
>>>
>>> Thanks.
>>>
>>> 2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago at stratio.com>:
>>>
>>>> Hi,
>>>>
>>>> I was testing your fix in several environments before replying,
>>>> unfortunately syslog-ng stills shows a huge RAM consumption in our use of
>>>> case.
>>>>
>>>> We have two wildcard-file sources defined in our syslog-ng conf file,
>>>> both of them have the same base-dir reference, but one source has a
>>>> filename-pattern and the other source has a different one (I think we can't
>>>> use the same source and several filename-patterns because of the lack of
>>>> regular expression support). We execute the deletion of obsolete files and
>>>> folder under base-dir once every 2 days, and we create a lot of new folders
>>>> and files constantly while using our platform, but the growth of these log
>>>> files (the ones affected by the filename-pattern) has been pretty low in
>>>> this test, about 10-20 msgs per second total.
>>>>
>>>> This is some data of the fix test:
>>>>
>>>> Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the
>>>> filename-pattern and 20000 folders under base-dir (not a single folder were
>>>> deleted yet so this is the expected behaviour of syslog-ng, and we find it
>>>> extremely unaffordable)
>>>>
>>>> One hour later, right before the cleaning process was executed,
>>>> syslog-ng was consuming 35G of RAM, and it didn't decrease after the
>>>> cleaning, not even a little, with just 380 files affected by the
>>>> filename-pattern and 2200 folders under base-dir.
>>>>
>>>> There is no message in the syslog-agent queue and all the messages are
>>>> received by the relay (which ingest messages in an elasticsearch)
>>>>
>>>> For the record, we do noticed a small RAM usage decrease in a previous
>>>> test in a lab environment when we deleted every folder & file under
>>>> base-dir (from 870M to 550M).
>>>>
>>>> It seems syslog-ng wildcard-source is not optimized when it has too
>>>> handle thousands of folders and filename-patterns under base-dir. Have you
>>>> ever test the wildcard-source with a use of case similar to ours?
>>>>
>>>> Can I send you any more information which may help to find out how to
>>>> optimize it? I wish I can help with the code but I have no idea of C
>>>> language :-(
>>>>
>>>> Thanks in advance.
>>>>
>>>>
>>>>
>>>>
>>>> 2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy at oneidentity.com>:
>>>>
>>>>> Hello,
>>>>>
>>>>> Thanks for the thorough investigation!
>>>>> We've checked it and found a memory leak in directory monitoring.
>>>>>
>>>>> I've pushed the fix to my fork and created a merge request about it:
>>>>> https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak
>>>>> https://github.com/balabit/syslog-ng/pull/2261
>>>>> Can you verify that this fixes your problem, please?
>>>>>
>>>>> > log-fifo-size value is 345000, so I assume it can't be a buffer
>>>>> situation since 345000 messages can't occupy 40-50 GB of memory.
>>>>> Well our default log-msg-size is 64kB so if all messages have a size
>>>>> of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
>>>>>
>>>>> Best Regards,
>>>>> Gabor
>>>>>
>>>>>
>>>>> On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago <
>>>>> jasantiago at stratio.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call
>>>>>> it "syslog-agent), which sends log messages to another syslog-ng acting as
>>>>>> a relay.
>>>>>>
>>>>>> I've noticed that syslog-agent instances RAM consumption keeps
>>>>>> increasing until they leave no free memory in the cluster (each server has
>>>>>> 64G of RAM). In my use case, new folders & files are created constantly
>>>>>> under base-dir folder, but every 2 days obsolete folders & files are
>>>>>> deleted. I assumed that syslog-ng would free some RAM every time those
>>>>>> folders & files are deleted, but it doesn't, not even if I run a
>>>>>> syslog-ng-ctl reload operation.
>>>>>>
>>>>>> log-fifo-size value is 345000, so I assume it can't be a buffer
>>>>>> situation since 345000 messages can't occupy 40-50 GB of memory.
>>>>>>
>>>>>> I've performed the following test to reproduce the situation in small
>>>>>> scale:
>>>>>>
>>>>>> - Launch a syslog-agent with a wildcard-file source reading from
>>>>>> "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M.
>>>>>> - Run a simple script to create complex folder hierarchy under
>>>>>> /tmp/test and some files with 5000 log messages to read from.
>>>>>> - Wait until syslog-agent RAM usage gets 1GB
>>>>>> - Stop script execution and wait until syslog-agent has send all logs
>>>>>> to relay.
>>>>>> - Delete everything under /tmp/test and execute syslog-ng-ctl reload
>>>>>> operation
>>>>>> - 24 hours after, syslog-agent RAM usage still is 1GB
>>>>>>
>>>>>> I've used heaptrack tool as a try to find a memory leak in syslog-ng,
>>>>>> you can see in the attached image that iv_list_empty function in iv_list.h
>>>>>> file is where most of the RAM usage is.
>>>>>>
>>>>>> How do I get syslog-ng to free RAM? Or is it a memory leak?
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> | Jose Angel Santiago
>>>>>>
>>>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>>>
>>>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>>>
>>>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>>>
>>>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>>>> <https://twitter.com/stratiobd>
>>>>>> <https://www.linkedin.com/company/stratiobd>
>>>>>> <https://www.youtube.com/c/StratioBD>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> | Jose Angel Santiago
>>>>
>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>
>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>
>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>
>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>> <https://twitter.com/stratiobd>
>>>> <https://www.linkedin.com/company/stratiobd>
>>>> <https://www.youtube.com/c/StratioBD>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> | Jose Angel Santiago
>>>
>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>
>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>
>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>
>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>> <https://twitter.com/stratiobd>
>>> <https://www.linkedin.com/company/stratiobd>
>>> <https://www.youtube.com/c/StratioBD>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
>
> | Jose Angel Santiago
>
> [image: Logo_signature2.png] <http://www.stratio.com/>
>
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>
> 28224 Pozuelo de Alarcón, Madrid, Spain
>
> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
> <https://twitter.com/stratiobd>
> <https://www.linkedin.com/company/stratiobd>
> <https://www.youtube.com/c/StratioBD>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180906/6bcc12d1/attachment-0001.html>
More information about the syslog-ng
mailing list