[syslog-ng] Excessive memory usage

[Patrick Hemmer]

Just to follow up on this, it does appear that the large buffers were
the issue. We switched to using disk buffering instead, and the memory
usage is much better now. The app starts out at 8mb used, goes up to
20mb over a few seconds, then drops back down to 8mb.

-Patrick

On 2018/1/16 08:00, Scheidler, Balázs wrote:
> Hi,
>
> we do release those buffers, but the libc malloc() may not.
>
> It could make sense to LD_PRELOAD the jemalloc allocator, that is
> better at returning pages to the system.
>
>
> -- 
> Bazsi
>
> On Thu, Jan 11, 2018 at 11:54 PM, Patrick Hemmer
> <syslogng at stormcloud9.net <mailto:syslogng at stormcloud9.net>> wrote:
>
>     We're seeing an issue on our systems where syslog-ng is consuming
>     an excessive amount of memory, approximately 380mb (RSS) right now
>     on one of our hosts.
>     I'm not sure what's causing this as our config is pretty minimal,
>     and there are no destinations unavailable which would cause it to
>     buffer. We do have `log-msg-size(65536)` and
>     `log-fifo-size(10000)` which might have caused it to buffer a
>     large amount some time in the past, but would it not release these
>     buffers?
>
>     Version 3.13.2 on CentOS/7 (RPM from
>     https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/
>     <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/>)
>
>     Config:
>     @version:3.13
>     @include "scl.conf"
>
>     # This configuration is meant to utilize systemd journald
>     options {
>       flush_lines(0);
>       time_reopen(10);
>       log-msg-size(65536);
>       log_fifo_size(10000);
>       chain_hostnames(off);
>       use_dns(no);
>       use_fqdn(yes);
>       keep_hostname(yes);
>       create_dirs(no);
>       stats_freq(3600);
>     };
>
>     source s_system { system(); };
>     source s_internal { internal(); };
>
>     destination d_journald {
>       unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n"));
>     };
>
>     log { source(s_internal); destination(d_journald); };
>
>     destination d_loghost {
>         syslog("syslog.ecom.chewy.com <http://syslog.ecom.chewy.com>"
>             frac-digits(3)
>             log-fifo-size(10000)
>         );
>     };
>
>     log { source(s_system); destination(d_loghost); };
>
>     Stats:
>
>     # syslog-ng-ctl stats
>     SourceName;SourceId;SourceInstance;State;Type;Number
>     dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025
>     dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242
>     dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1
>     dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216
>     destination;d_journald;;a;processed;1997
>     src.internal;s_internal#0;;a;processed;1997
>     src.internal;s_internal#0;;a;stamp;1515709115
>     center;;received;a;processed;1997
>     destination;d_loghost;;a;processed;489030242
>     source;s_internal;;a;processed;1997
>     center;;queued;a;processed;489032239
>     global;scratch_buffers_count;;a;queued;25769803782
>     dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0
>     dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845
>     dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0
>     dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845
>     global;payload_reallocs;;a;processed;2932894697
>     src.journald;;journal;a;processed;489030242
>     src.journald;;journal;a;stamp;1515710081
>     global;sdata_updates;;a;processed;0
>     global;scratch_buffers_bytes;;a;queued;0
>     global;msg_clones;;a;processed;135
>     global;internal_queue_length;;a;processed;0
>
>     Thanks
>
>     -Patrick
>
>
>
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
>     Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     <http://www.balabit.com/support/documentation/?product=syslog-ng>
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>     <http://www.balabit.com/wiki/syslog-ng-faq>
>
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180125/049f95ef/attachment-0001.html>