[syslog-ng] syslog-ng never frees RAM

Jose Angel Santiago jasantiago at stratio.com
Mon Sep 10 06:25:25 UTC 2018


Hi,

We've tested these 2 fixes along with the previous one during this weekend
in our demo environment and now the RAM usage seems to be pretty fair, so I
guess fixes are working great!

Thanks for your quick & effective response.

2018-09-06 16:45 GMT+02:00 Nemes, Antal <antal.nemes at oneidentity.com>:

>   Hi,
>
> We found another leak related to monitor-method(poll). When the tracked
> directory is deleted, glib allocates a GError structure for "Error opening
> directory \'...\': No such file or directory", that was not freed by
> syslog-ng. This can cause considerable memory leak when high number of
> directories tracked and deleted. We created a pull request to fix it: here
> you can track the progress: https://github.com/balabit/syslog-ng/pull/2267
>
> Br,
>   Antal
>
> On Thu, Sep 6, 2018 at 1:15 PM Jose Angel Santiago <jasantiago at stratio.com>
> wrote:
>
>> Hi,
>>
>> I had to restart syslog-ng process because it was affecting other
>> services in the same host.
>>
>> I've checked another host with was running with the fix too, it has 23000
>> files monitored and 38700 folders under base-dir, and now it's starting to
>> eat the RAM, the cleaning process will be executed in about an hour (I'm
>> sorry I can't force to do it before)
>>
>> PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>> 96 root      20   0  *5.609g 4.274g* 0.006g R *101.3*  *6.8 *  1263:46
>> syslog-ng
>>
>> Trace of pmap:
>>
>> 96:   /syslog-ng/exec/sbin/syslog-ng -F --no-caps -f
>> /etc/syslog-ng/syslog-ng.conf -p /var/syslog-ng/syslog-ng.pid -R
>> /var/syslog-ng/syslog-ng.persist -c /var/syslog-ng/syslog-ng.ctl
>> 0000000000400000     12K r-x-- syslog-ng
>> 0000000000602000      4K r---- syslog-ng
>> 0000000000603000      4K rw--- syslog-ng
>> 00000000015bc000 5689360K rw---   [ anon ]
>> 00007ff9c8000000  19588K rw---   [ anon ]
>> 00007ff9c9321000  45948K -----   [ anon ]
>> 00007ff9cc000000   4288K rw---   [ anon ]
>> 00007ff9cc430000  61248K -----   [ anon ]
>> 00007ff9d0000000  26844K rw---   [ anon ]
>> 00007ff9d1a37000  38692K -----   [ anon ]
>> 00007ff9d4000000  10280K rw---   [ anon ]
>> 00007ff9d4a0a000  55256K -----   [ anon ]
>> 00007ff9dc000000  64604K rw---   [ anon ]
>> 00007ff9dff17000    932K -----   [ anon ]
>> 00007ff9e0000000  65508K rw---   [ anon ]
>> 00007ff9e3ff9000     28K -----   [ anon ]
>> 00007ff9e8000000   4480K rw---   [ anon ]
>> 00007ff9e8460000  61056K -----   [ anon ]
>> 00007ff9f0000000  65488K rw---   [ anon ]
>> 00007ff9f3ff4000     48K -----   [ anon ]
>> 00007ff9f8000000  14716K rw---   [ anon ]
>> 00007ff9f8e5f000  50820K -----   [ anon ]
>> 00007ffa00000000  18252K rw---   [ anon ]
>> 00007ffa011d3000  47284K -----   [ anon ]
>> 00007ffa08000000  65536K rw---   [ anon ]
>> 00007ffa0c000000  30460K rw---   [ anon ]
>> 00007ffa0ddbf000  35076K -----   [ anon ]
>> 00007ffa10000000  65484K rw---   [ anon ]
>> 00007ffa13ff3000     52K -----   [ anon ]
>> 00007ffa167fd000      4K -----   [ anon ]
>> 00007ffa167fe000   8192K rw---   [ anon ]
>> 00007ffa18000000   7920K rw---   [ anon ]
>> 00007ffa187bc000  57616K -----   [ anon ]
>> 00007ffa1cffa000      4K -----   [ anon ]
>> 00007ffa1cffb000   8192K rw---   [ anon ]
>> 00007ffa1f7ff000      4K -----   [ anon ]
>> 00007ffa1f800000   8192K rw---   [ anon ]
>> 00007ffa20000000  65536K rw---   [ anon ]
>> 00007ffa24000000  14764K rw---   [ anon ]
>> 00007ffa24e6b000  50772K -----   [ anon ]
>> 00007ffa28000000  24348K rw---   [ anon ]
>> 00007ffa297c7000  41188K -----   [ anon ]
>> 00007ffa2c000000  40916K rw---   [ anon ]
>> 00007ffa2e7f5000  24620K -----   [ anon ]
>> 00007ffa30000000  17684K rw---   [ anon ]
>> 00007ffa31145000  47852K -----   [ anon ]
>> 00007ffa34000000  10988K rw---   [ anon ]
>> 00007ffa34abb000  54548K -----   [ anon ]
>> 00007ffa38000000  20464K rw---   [ anon ]
>> 00007ffa393fc000  45072K -----   [ anon ]
>> 00007ffa3c316000      4K -----   [ anon ]
>> 00007ffa3c317000   8192K rw---   [ anon ]
>> 00007ffa3ea78000      4K -----   [ anon ]
>> 00007ffa3ea79000   8192K rw---   [ anon ]
>> 00007ffa3fbee000   7360K rw-s- syslog-ng.persist
>> 00007ffa4031e000     92K r-x-- libresolv-2.23.so
>> 00007ffa40335000   2048K ----- libresolv-2.23.so
>> 00007ffa40535000      4K r---- libresolv-2.23.so
>> 00007ffa40536000      4K rw--- libresolv-2.23.so
>> 00007ffa40537000      8K rw---   [ anon ]
>> 00007ffa40539000     20K r-x-- libnss_dns-2.23.so
>> 00007ffa4053e000   2048K ----- libnss_dns-2.23.so
>> 00007ffa4073e000      4K r---- libnss_dns-2.23.so
>> 00007ffa4073f000      4K rw--- libnss_dns-2.23.so
>> 00007ffa40740000   3328K rw---   [ anon ]
>> 00007ffa40ac0000    256K rw---   [ anon ]
>> 00007ffa40b40000    512K rw---   [ anon ]
>> 00007ffa40c00000    256K rw---   [ anon ]
>> 00007ffa40c6c000    256K rw---   [ anon ]
>> 00007ffa40d6c000     44K r-x-- libsyslogformat.so
>> 00007ffa40d77000   2044K ----- libsyslogformat.so
>> 00007ffa40f76000      4K r---- libsyslogformat.so
>> 00007ffa40f77000      4K rw--- libsyslogformat.so
>> 00007ffa40f78000   4612K rw---   [ anon ]
>> 00007ffa413f9000     24K r-x-- _hashlib.x86_64-linux-gnu.so
>> 00007ffa413ff000   2044K ----- _hashlib.x86_64-linux-gnu.so
>> 00007ffa415fe000      4K r---- _hashlib.x86_64-linux-gnu.so
>> 00007ffa415ff000      4K rw--- _hashlib.x86_64-linux-gnu.so
>> 00007ffa41600000    256K rw---   [ anon ]
>> 00007ffa41640000     48K r-x-- _json.x86_64-linux-gnu.so
>> 00007ffa4164c000   2044K ----- _json.x86_64-linux-gnu.so
>> 00007ffa4184b000      4K r---- _json.x86_64-linux-gnu.so
>> 00007ffa4184c000      4K rw--- _json.x86_64-linux-gnu.so
>> 00007ffa4184d000     84K r-x-- _ssl.x86_64-linux-gnu.so
>> 00007ffa41862000   2044K ----- _ssl.x86_64-linux-gnu.so
>> 00007ffa41a61000      4K r---- _ssl.x86_64-linux-gnu.so
>> 00007ffa41a62000     16K rw--- _ssl.x86_64-linux-gnu.so
>> 00007ffa41a66000    768K rw---   [ anon ]
>> 00007ffa41b57000    256K rw---   [ anon ]
>> 00007ffa41b97000   1056K r-x-- libm-2.23.so
>> 00007ffa41c9f000   2044K ----- libm-2.23.so
>> 00007ffa41e9e000      4K r---- libm-2.23.so
>> 00007ffa41e9f000      4K rw--- libm-2.23.so
>> 00007ffa41ea0000      8K r-x-- libutil-2.23.so
>> 00007ffa41ea2000   2044K ----- libutil-2.23.so
>> 00007ffa420a1000      4K r---- libutil-2.23.so
>> 00007ffa420a2000      4K rw--- libutil-2.23.so
>> 00007ffa420a3000    100K r-x-- libz.so.1.2.8
>> 00007ffa420bc000   2044K ----- libz.so.1.2.8
>> 00007ffa422bb000      4K r---- libz.so.1.2.8
>> 00007ffa422bc000      4K rw--- libz.so.1.2.8
>> 00007ffa422bd000   3016K r-x-- libpython2.7.so.1.0
>> 00007ffa425af000   2048K ----- libpython2.7.so.1.0
>> 00007ffa427af000      8K r---- libpython2.7.so.1.0
>> 00007ffa427b1000    476K rw--- libpython2.7.so.1.0
>> 00007ffa42828000    140K rw---   [ anon ]
>> 00007ffa4284b000     72K r-x-- libmod-python.so
>> 00007ffa4285d000   2044K ----- libmod-python.so
>> 00007ffa42a5c000      4K r---- libmod-python.so
>> 00007ffa42a5d000      4K rw--- libmod-python.so
>> 00007ffa42a5e000      4K rw---   [ anon ]
>> 00007ffa42a5f000     32K r-x-- libwrap.so.0.7.6
>> 00007ffa42a67000   2044K ----- libwrap.so.0.7.6
>> 00007ffa42c66000      4K r---- libwrap.so.0.7.6
>> 00007ffa42c67000      4K rw--- libwrap.so.0.7.6
>> 00007ffa42c68000      4K rw---   [ anon ]
>> 00007ffa42c69000     88K r-x-- libnet.so.1.7.0
>> 00007ffa42c7f000   2044K ----- libnet.so.1.7.0
>> 00007ffa42e7e000      4K r---- libnet.so.1.7.0
>> 00007ffa42e7f000      4K rw--- libnet.so.1.7.0
>> 00007ffa42e80000      8K rw---   [ anon ]
>> 00007ffa42e82000    132K r-x-- libafsocket.so
>> 00007ffa42ea3000   2048K ----- libafsocket.so
>> 00007ffa430a3000      4K r---- libafsocket.so
>> 00007ffa430a4000      8K rw--- libafsocket.so
>> 00007ffa430a6000      4K rw---   [ anon ]
>> 00007ffa430a7000    108K r-x-- libaffile.so
>> 00007ffa430c2000   2044K ----- libaffile.so
>> 00007ffa432c1000      4K r---- libaffile.so
>> 00007ffa432c2000      4K rw--- libaffile.so
>> 00007ffa432c3000      4K rw---   [ anon ]
>> 00007ffa432c4000      8K r-x-- libconfgen.so
>> 00007ffa432c6000   2044K ----- libconfgen.so
>> 00007ffa434c5000      4K r---- libconfgen.so
>> 00007ffa434c6000      4K rw--- libconfgen.so
>> 00007ffa434c7000     36K r-x-- libbasicfuncs.so
>> 00007ffa434d0000   2048K ----- libbasicfuncs.so
>> 00007ffa436d0000      4K r---- libbasicfuncs.so
>> 00007ffa436d1000      8K rw--- libbasicfuncs.so
>> 00007ffa436d3000     32K r-x-- libappmodel.so
>> 00007ffa436db000   2048K ----- libappmodel.so
>> 00007ffa438db000      4K r---- libappmodel.so
>> 00007ffa438dc000      4K rw--- libappmodel.so
>> 00007ffa438dd000     40K r-x-- libjson-c.so.2.0.0
>> 00007ffa438e7000   2044K ----- libjson-c.so.2.0.0
>> 00007ffa43ae6000      4K r---- libjson-c.so.2.0.0
>> 00007ffa43ae7000      4K rw--- libjson-c.so.2.0.0
>> 00007ffa43ae8000     40K r-x-- libjson-plugin.so
>> 00007ffa43af2000   2048K ----- libjson-plugin.so
>> 00007ffa43cf2000      4K r---- libjson-plugin.so
>> 00007ffa43cf3000      4K rw--- libjson-plugin.so
>> 00007ffa43cf4000     44K r-x-- libnss_nis-2.23.so
>> 00007ffa43cff000   2044K ----- libnss_nis-2.23.so
>> 00007ffa43efe000      4K r---- libnss_nis-2.23.so
>> 00007ffa43eff000      4K rw--- libnss_nis-2.23.so
>> 00007ffa43f00000     88K r-x-- libnsl-2.23.so
>> 00007ffa43f16000   2044K ----- libnsl-2.23.so
>> 00007ffa44115000      4K r---- libnsl-2.23.so
>> 00007ffa44116000      4K rw--- libnsl-2.23.so
>> 00007ffa44117000      8K rw---   [ anon ]
>> 00007ffa44119000     32K r-x-- libnss_compat-2.23.so
>> 00007ffa44121000   2044K ----- libnss_compat-2.23.so
>> 00007ffa44320000      4K r---- libnss_compat-2.23.so
>> 00007ffa44321000      4K rw--- libnss_compat-2.23.so
>> 00007ffa44322000     44K r-x-- libnss_files-2.23.so
>> 00007ffa4432d000   2044K ----- libnss_files-2.23.so
>> 00007ffa4452c000      4K r---- libnss_files-2.23.so
>> 00007ffa4452d000      4K rw--- libnss_files-2.23.so
>> 00007ffa4452e000     24K rw---   [ anon ]
>> 00007ffa44534000     72K r-x-- libgpg-error.so.0.17.0
>> 00007ffa44546000   2048K ----- libgpg-error.so.0.17.0
>> 00007ffa44746000      4K r---- libgpg-error.so.0.17.0
>> 00007ffa44747000      4K rw--- libgpg-error.so.0.17.0
>> 00007ffa44748000    864K r-x-- libgcrypt.so.20.0.5
>> 00007ffa44820000   2044K ----- libgcrypt.so.20.0.5
>> 00007ffa44a1f000      4K r---- libgcrypt.so.20.0.5
>> 00007ffa44a20000     32K rw--- libgcrypt.so.20.0.5
>> 00007ffa44a28000      4K rw---   [ anon ]
>> 00007ffa44a29000    132K r-x-- liblzma.so.5.0.0
>> 00007ffa44a4a000   2044K ----- liblzma.so.5.0.0
>> 00007ffa44c49000      4K r---- liblzma.so.5.0.0
>> 00007ffa44c4a000      4K rw--- liblzma.so.5.0.0
>> 00007ffa44c4b000    124K r-x-- libselinux.so.1
>> 00007ffa44c6a000   2044K ----- libselinux.so.1
>> 00007ffa44e69000      4K r---- libselinux.so.1
>> 00007ffa44e6a000      4K rw--- libselinux.so.1
>> 00007ffa44e6b000      8K rw---   [ anon ]
>> 00007ffa44e6d000   2152K r-x-- libcrypto.so.1.0.0
>> 00007ffa45087000   2044K ----- libcrypto.so.1.0.0
>> 00007ffa45286000    112K r---- libcrypto.so.1.0.0
>> 00007ffa452a2000     48K rw--- libcrypto.so.1.0.0
>> 00007ffa452ae000     12K rw---   [ anon ]
>> 00007ffa452b1000    376K r-x-- libssl.so.1.0.0
>> 00007ffa4530f000   2048K ----- libssl.so.1.0.0
>> 00007ffa4550f000     16K r---- libssl.so.1.0.0
>> 00007ffa45513000     28K rw--- libssl.so.1.0.0
>> 00007ffa4551a000     12K r-x-- libsecret-storage.so.0.0.0
>> 00007ffa4551d000   2044K ----- libsecret-storage.so.0.0.0
>> 00007ffa4571c000      4K r---- libsecret-storage.so.0.0.0
>> 00007ffa4571d000      4K rw--- libsecret-storage.so.0.0.0
>> 00007ffa4571e000     12K r-x-- libdl-2.23.so
>> 00007ffa45721000   2044K ----- libdl-2.23.so
>> 00007ffa45920000      4K r---- libdl-2.23.so
>> 00007ffa45921000      4K rw--- libdl-2.23.so
>> 00007ffa45922000    440K r-x-- libpcre.so.3.13.2
>> 00007ffa45990000   2048K ----- libpcre.so.3.13.2
>> 00007ffa45b90000      4K r---- libpcre.so.3.13.2
>> 00007ffa45b91000      4K rw--- libpcre.so.3.13.2
>> 00007ffa45b92000     16K r-x-- libcap.so.2.24
>> 00007ffa45b96000   2048K ----- libcap.so.2.24
>> 00007ffa45d96000      4K r---- libcap.so.2.24
>> 00007ffa45d97000      4K rw--- libcap.so.2.24
>> 00007ffa45d98000     12K r-x-- libevtlog-3.17.so.0.0.0
>> 00007ffa45d9b000   2048K ----- libevtlog-3.17.so.0.0.0
>> 00007ffa45f9b000      4K r---- libevtlog-3.17.so.0.0.0
>> 00007ffa45f9c000      4K rw--- libevtlog-3.17.so.0.0.0
>> 00007ffa45f9d000      4K r-x-- libgthread-2.0.so.0.4800.2
>> 00007ffa45f9e000   2044K ----- libgthread-2.0.so.0.4800.2
>> 00007ffa4619d000      4K r---- libgthread-2.0.so.0.4800.2
>> 00007ffa4619e000      4K rw--- libgthread-2.0.so.0.4800.2
>> 00007ffa4619f000     12K r-x-- libgmodule-2.0.so.0.4800.2
>> 00007ffa461a2000   2044K ----- libgmodule-2.0.so.0.4800.2
>> 00007ffa463a1000      4K r---- libgmodule-2.0.so.0.4800.2
>> 00007ffa463a2000      4K rw--- libgmodule-2.0.so.0.4800.2
>> 00007ffa463a3000     28K r-x-- librt-2.23.so
>> 00007ffa463aa000   2044K ----- librt-2.23.so
>> 00007ffa465a9000      4K r---- librt-2.23.so
>> 00007ffa465aa000      4K rw--- librt-2.23.so
>> 00007ffa465ab000   1792K r-x-- libc-2.23.so
>> 00007ffa4676b000   2044K ----- libc-2.23.so
>> 00007ffa4696a000     16K r---- libc-2.23.so
>> 00007ffa4696e000      8K rw--- libc-2.23.so
>> 00007ffa46970000     16K rw---   [ anon ]
>> 00007ffa46974000     96K r-x-- libpthread-2.23.so
>> 00007ffa4698c000   2044K ----- libpthread-2.23.so
>> 00007ffa46b8b000      4K r---- libpthread-2.23.so
>> 00007ffa46b8c000      4K rw--- libpthread-2.23.so
>> 00007ffa46b8d000     16K rw---   [ anon ]
>> 00007ffa46b91000   1084K r-x-- libglib-2.0.so.0.4800.2
>> 00007ffa46ca0000   2044K ----- libglib-2.0.so.0.4800.2
>> 00007ffa46e9f000      4K r---- libglib-2.0.so.0.4800.2
>> 00007ffa46ea0000      4K rw--- libglib-2.0.so.0.4800.2
>> 00007ffa46ea1000      4K rw---   [ anon ]
>> 00007ffa46ea2000    768K r-x-- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa46f62000   2044K ----- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa47161000     16K r---- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa47165000     24K rw--- libsyslog-ng-3.17.so.0.0.0
>> 00007ffa4716b000      8K rw---   [ anon ]
>> 00007ffa4716d000    152K r-x-- ld-2.23.so
>> 00007ffa471ba000    256K rw---   [ anon ]
>> 00007ffa472a2000    256K rw---   [ anon ]
>> 00007ffa472e2000     64K rwx--   [ anon ]
>> 00007ffa472f2000     28K r--s- gconv-modules.cache
>> 00007ffa472f9000     36K rw---   [ anon ]
>> 00007ffa47302000    512K r-x-- libsystemd.so.0.14.0
>> 00007ffa47382000     12K r---- libsystemd.so.0.14.0
>> 00007ffa47385000      4K rw--- libsystemd.so.0.14.0
>> 00007ffa47386000     20K rw---   [ anon ]
>> 00007ffa47390000      8K rw---   [ anon ]
>> 00007ffa47392000      4K r---- ld-2.23.so
>> 00007ffa47393000      4K rw--- ld-2.23.so
>> 00007ffa47394000      4K rw---   [ anon ]
>> 00007ffe81489000    132K rw---   [ stack ]
>> 00007ffe81564000      8K r-x--   [ anon ]
>> ffffffffff600000      4K r-x--   [ anon ]
>>  total          7226528K
>>
>>
>> Configuration:
>>
>> @version: 3.16.1
>> @module mod-java
>> @include "/etc/syslog-ng/scl.conf"
>>
>> options {
>>     chain_hostnames(off);
>>     use-dns(no);
>>     use-fqdn(no);
>>     owner("syslog-ng");
>>     group("stratio");
>>     perm(0640);
>>     stats_freq(0);
>>     bad-hostname("^gconfd$");
>>     flush-lines(100);
>>     log-fifo-size(345000);
>> };
>>
>> ########################
>> # Filters
>> ########################
>> # Here's come the filter options. With this rules, we can set which
>> # message go where.
>>
>> filter f_dbg { level(debug); };
>> filter f_info { level(info); };
>> filter f_notice { level(notice); };
>> filter f_warn { level(warn); };
>> filter f_err { level(err); };
>> filter f_crit { level(crit .. emerg); };
>>
>> filter f_debug { level(debug) and not facility(auth, authpriv, news,
>> mail); };
>> filter f_error { level(err .. emerg) ; };
>> filter f_messages { level(info,notice,warn) and
>>                     not facility(auth,authpriv,cron,daemon,mail,news); };
>>
>> filter f_auth { facility(auth, authpriv) and not filter(f_debug); };
>> filter f_cron { facility(cron) and not filter(f_debug); };
>> filter f_daemon { facility(daemon) and not filter(f_debug); };
>> filter f_kern { facility(kern) and not filter(f_debug); };
>> filter f_lpr { facility(lpr) and not filter(f_debug); };
>> filter f_local { facility(local0, local1, local3, local4, local5,
>>                         local6, local7) and not filter(f_debug); };
>> filter f_mail { facility(mail) and not filter(f_debug); };
>> filter f_news { facility(news) and not filter(f_debug); };
>> filter f_syslog3 { not facility(auth, authpriv, mail) and not
>> filter(f_debug); };
>> filter f_user { facility(user) and not filter(f_debug); };
>>
>> filter f_text_only {
>>   not match(
>>     ".*[\\x80-\\xFF].*"
>>     value("MESSAGE")
>>   );
>> };
>>
>> ###
>> # Sources
>>
>> source s_stdout_files { wildcard-file(
>>     base-dir("/var/lib/slaves/")
>>     filename-pattern("stdout")
>>     max-files(3400)
>>     follow-freq(0.5)
>>     monitor-method(poll)
>>     log-fetch-limit(100)
>>     log-iw-size(345000)
>>     flags(no-parse)
>>     recursive(yes)
>>     multi-line-mode(indented)
>> ); };
>>
>> source s_stderr_files { wildcard-file(
>>     base-dir("/var/lib/slaves/")
>>     filename-pattern("stderr")
>>     max-files(3400)
>>     follow-freq(0.5)
>>     monitor-method(poll)
>>     log-fetch-limit(100)
>>     log-iw-size(345000)
>>     flags(no-parse)
>>     recursive(yes)
>>     multi-line-mode(indented)
>> ); };
>>
>>
>> Thanks.
>>
>>
>> 2018-09-06 12:59 GMT+02:00 Nemes, Antal <antal.nemes at oneidentity.com>:
>>
>>>   Hi,
>>>
>>> The same free policy cannot work with _handle_file_deleted, because we
>>> cannot free the readers immediately. There is a race between file delete
>>> and syslog-ng reading the data. It can happen that one deletes a file
>>> before syslog-ng reads all its contents, which would cause message loss. In
>>> that case we just signal the deletion event to the reader
>>> (log_pipe_notify(&reader->super, NC_FILE_DELETED, NULL); and when the
>>> reader reads EOF at a later time, that's when the reader object is freed.
>>>
>>> Can you send the output of pmap <pid of syslog-ng> with the leak? That
>>> may give us a clue where to look with the leak.
>>>
>>> Also can you send your configuration, or at least the part about the
>>> wildcard filesources? The reproduction might depend on max_files or
>>> monitor_method parameter, or maybe other.
>>>
>>> Thanks,
>>>   Antal
>>>
>>>
>>> On Thu, Sep 6, 2018 at 12:00 PM Jose Angel Santiago <
>>> jasantiago at stratio.com> wrote:
>>>
>>>> Reviewing your commit I've found out that the method
>>>> *_handle_file_deleted* seems not to free memory when a monitored file
>>>> has been deleted, according to the changes you made in
>>>> *_handler_handler_directory_deleted.* Could it be the cause of the
>>>> memory leak?
>>>>
>>>> Thanks.
>>>>
>>>> 2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago at stratio.com>
>>>> :
>>>>
>>>>> Hi,
>>>>>
>>>>> I was testing your fix in several environments before replying,
>>>>> unfortunately syslog-ng stills shows a huge RAM consumption in our use of
>>>>> case.
>>>>>
>>>>> We have two wildcard-file sources defined in our syslog-ng conf file,
>>>>> both of them have the same base-dir reference, but one source has a
>>>>> filename-pattern and the other source has a different one (I think we can't
>>>>> use the same source and several filename-patterns because of the lack of
>>>>> regular expression support). We execute the deletion of obsolete files and
>>>>> folder under base-dir once every 2 days, and we create a lot of new folders
>>>>> and files constantly while using our platform, but the growth of these log
>>>>> files (the ones affected by the filename-pattern) has been pretty low in
>>>>> this test, about 10-20 msgs per second total.
>>>>>
>>>>> This is some data of the fix test:
>>>>>
>>>>> Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the
>>>>> filename-pattern and 20000 folders under base-dir (not a single folder were
>>>>> deleted yet so this is the expected behaviour of syslog-ng, and we find it
>>>>> extremely unaffordable)
>>>>>
>>>>> One hour later, right before the cleaning process was executed,
>>>>> syslog-ng was consuming 35G of RAM, and it didn't decrease after the
>>>>> cleaning, not even a little, with just 380 files affected by the
>>>>> filename-pattern and 2200 folders under base-dir.
>>>>>
>>>>> There is no message in the syslog-agent queue and all the messages are
>>>>> received by the relay (which ingest messages in an elasticsearch)
>>>>>
>>>>> For the record, we do noticed a small RAM usage decrease in a previous
>>>>> test in a lab environment when we deleted every folder & file under
>>>>> base-dir (from 870M to 550M).
>>>>>
>>>>> It seems syslog-ng wildcard-source is not optimized when it has too
>>>>> handle thousands of folders and filename-patterns under base-dir. Have you
>>>>> ever test the wildcard-source with a use of case similar to ours?
>>>>>
>>>>> Can I send you any more information which may help to find out how to
>>>>> optimize it? I wish I can help with the code but I have no idea of C
>>>>> language :-(
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy at oneidentity.com>:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Thanks for the thorough investigation!
>>>>>> We've checked it and found a memory leak in directory monitoring.
>>>>>>
>>>>>> I've pushed the fix to my fork and created a merge request about it:
>>>>>> https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak
>>>>>> https://github.com/balabit/syslog-ng/pull/2261
>>>>>> Can you verify that this fixes your problem, please?
>>>>>>
>>>>>> > log-fifo-size value is 345000, so I assume it can't be a buffer
>>>>>> situation since 345000 messages can't occupy 40-50 GB of memory.
>>>>>> Well our default log-msg-size is 64kB so if all messages have a size
>>>>>> of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
>>>>>>
>>>>>> Best Regards,
>>>>>> Gabor
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago <
>>>>>> jasantiago at stratio.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call
>>>>>>> it "syslog-agent), which sends log messages to another syslog-ng acting as
>>>>>>> a relay.
>>>>>>>
>>>>>>> I've noticed that syslog-agent instances RAM consumption keeps
>>>>>>> increasing until they leave no free memory in the cluster (each server has
>>>>>>> 64G of RAM). In my use case, new folders & files are created constantly
>>>>>>> under base-dir folder, but every 2 days obsolete folders & files are
>>>>>>> deleted. I assumed that syslog-ng would free some RAM every time those
>>>>>>> folders & files are deleted, but it doesn't, not even if I run a
>>>>>>> syslog-ng-ctl reload operation.
>>>>>>>
>>>>>>> log-fifo-size value is 345000, so I assume it can't be a buffer
>>>>>>> situation since 345000 messages can't occupy 40-50 GB of memory.
>>>>>>>
>>>>>>> I've performed the following test to reproduce the situation in
>>>>>>> small scale:
>>>>>>>
>>>>>>> - Launch a syslog-agent with a wildcard-file source reading from
>>>>>>> "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M.
>>>>>>> - Run a simple script to create complex folder hierarchy under
>>>>>>> /tmp/test and some files with 5000 log messages to read from.
>>>>>>> - Wait until syslog-agent RAM usage gets 1GB
>>>>>>> - Stop script execution and wait until syslog-agent has send all
>>>>>>> logs to relay.
>>>>>>> - Delete everything under /tmp/test and execute syslog-ng-ctl reload
>>>>>>> operation
>>>>>>> - 24 hours after, syslog-agent RAM usage still is 1GB
>>>>>>>
>>>>>>> I've used heaptrack tool as a try to find a memory leak in
>>>>>>> syslog-ng, you can see in the attached image that iv_list_empty function in
>>>>>>> iv_list.h file is where most of the RAM usage is.
>>>>>>>
>>>>>>> How do I get syslog-ng to free RAM? Or is it a memory leak?
>>>>>>>
>>>>>>> Thanks in advance.
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> | Jose Angel Santiago
>>>>>>>
>>>>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>>>>
>>>>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>>>>
>>>>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>>>>
>>>>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>>>>> <https://twitter.com/stratiobd>
>>>>>>> <https://www.linkedin.com/company/stratiobd>
>>>>>>> <https://www.youtube.com/c/StratioBD>
>>>>>>> ____________________________________________________________
>>>>>>> __________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>>>>> product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ____________________________________________________________
>>>>>> __________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>>>> product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> | Jose Angel Santiago
>>>>>
>>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>>
>>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>>
>>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>>
>>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>>> <https://twitter.com/stratiobd>
>>>>> <https://www.linkedin.com/company/stratiobd>
>>>>> <https://www.youtube.com/c/StratioBD>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> | Jose Angel Santiago
>>>>
>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>
>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>
>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>
>>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>>> <https://twitter.com/stratiobd>
>>>> <https://www.linkedin.com/company/stratiobd>
>>>> <https://www.youtube.com/c/StratioBD>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>> product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?
>>> product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> --
>>
>> | Jose Angel Santiago
>>
>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>
>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>
>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>
>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>> <https://twitter.com/stratiobd>
>> <https://www.linkedin.com/company/stratiobd>
>> <https://www.youtube.com/c/StratioBD>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>


-- 

| Jose Angel Santiago

[image: Logo_signature2.png] <http://www.stratio.com/>

Vía de las dos Castillas, 33, Ática 4, 3ª Planta

28224 Pozuelo de Alarcón, Madrid, Spain

+34 918 286 473 | www.stratio.com
<https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd>
<https://www.youtube.com/c/StratioBD>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180910/841c6c20/attachment-0001.html>


More information about the syslog-ng mailing list