Just to follow up on this, it does appear that the large buffers were the issue. We switched to using disk buffering instead, and the memory usage is much better now. The app starts out at 8mb used, goes up to 20mb over a few seconds, then drops back down to 8mb.

-Patrick

On 2018/1/16 08:00, Scheidler, Balázs wrote:

Hi,

we do release those buffers, but the libc malloc() may not.

It could make sense to LD_PRELOAD the jemalloc allocator, that is better at returning pages to the system.

--
Bazsi

On Thu, Jan 11, 2018 at 11:54 PM, Patrick Hemmer <syslogng@stormcloud9.net> wrote:

We're seeing an issue on our systems where syslog-ng is consuming an excessive amount of memory, approximately 380mb (RSS) right now on one of our hosts.
I'm not sure what's causing this as our config is pretty minimal, and there are no destinations unavailable which would cause it to buffer. We do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have caused it to buffer a large amount some time in the past, but would it not release these buffers?

Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/)

Config:
@version:3.13
@include "scl.conf"

# This configuration is meant to utilize systemd journald
options {
  flush_lines(0);
  time_reopen(10);
  log-msg-size(65536);
  log_fifo_size(10000);
  chain_hostnames(off);
  use_dns(no);
  use_fqdn(yes);
  keep_hostname(yes);
  create_dirs(no);
  stats_freq(3600);
};

source s_system { system(); };
source s_internal { internal(); };

destination d_journald {
  unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n"));
};

log { source(s_internal); destination(d_journald); };

destination d_loghost {
    syslog("syslog.ecom.chewy.com"
        frac-digits(3)
        log-fifo-size(10000)
    );
};

log { source(s_system); destination(d_loghost); };

Stats:

# syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216
destination;d_journald;;a;processed;1997
src.internal;s_internal#0;;a;processed;1997
src.internal;s_internal#0;;a;stamp;1515709115
center;;received;a;processed;1997
destination;d_loghost;;a;processed;489030242
source;s_internal;;a;processed;1997
center;;queued;a;processed;489032239
global;scratch_buffers_count;;a;queued;25769803782
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845
global;payload_reallocs;;a;processed;2932894697
src.journald;;journal;a;processed;489030242
src.journald;;journal;a;stamp;1515710081
global;sdata_updates;;a;processed;0
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;135
global;internal_queue_length;;a;processed;0

Thanks

-Patrick

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq