[syslog-ng] syslog-ng never frees RAM

[Nagy, Gábor]

Hello,

Thanks for the thorough investigation!
We've checked it and found a memory leak in directory monitoring.

I've pushed the fix to my fork and created a merge request about it:
https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak
https://github.com/balabit/syslog-ng/pull/2261
Can you verify that this fixes your problem, please?

> log-fifo-size value is 345000, so I assume it can't be a buffer situation
since 345000 messages can't occupy 40-50 GB of memory.
Well our default log-msg-size is 64kB so if all messages have a size of
log-msg-size then it can be around ~21GB when syslog-ng is buffering.

Best Regards,
Gabor


On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago <jasantiago at stratio.com>
wrote:

> Hi,
>
> I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it
> "syslog-agent), which sends log messages to another syslog-ng acting as a
> relay.
>
> I've noticed that syslog-agent instances RAM consumption keeps increasing
> until they leave no free memory in the cluster (each server has 64G of
> RAM). In my use case, new folders & files are created constantly under
> base-dir folder, but every 2 days obsolete folders & files are deleted. I
> assumed that syslog-ng would free some RAM every time those folders & files
> are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload
> operation.
>
> log-fifo-size value is 345000, so I assume it can't be a buffer situation
> since 345000 messages can't occupy 40-50 GB of memory.
>
> I've performed the following test to reproduce the situation in small
> scale:
>
> - Launch a syslog-agent with a wildcard-file source reading from
> "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M.
> - Run a simple script to create complex folder hierarchy under /tmp/test
> and some files with 5000 log messages to read from.
> - Wait until syslog-agent RAM usage gets 1GB
> - Stop script execution and wait until syslog-agent has send all logs to
> relay.
> - Delete everything under /tmp/test and execute syslog-ng-ctl reload
> operation
> - 24 hours after, syslog-agent RAM usage still is 1GB
>
> I've used heaptrack tool as a try to find a memory leak in syslog-ng, you
> can see in the attached image that iv_list_empty function in iv_list.h file
> is where most of the RAM usage is.
>
> How do I get syslog-ng to free RAM? Or is it a memory leak?
>
> Thanks in advance.
>
>
> --
>
> | Jose Angel Santiago
>
> [image: Logo_signature2.png] <http://www.stratio.com/>
>
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>
> 28224 Pozuelo de Alarcón, Madrid, Spain
>
> +34 918 286 473 | www.stratio.com
> <https://twitter.com/stratiobd>
> <https://www.linkedin.com/company/stratiobd>
> <https://www.youtube.com/c/StratioBD>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180904/9cc70940/attachment.html>