[syslog-ng] Excessive memory usage

[Patrick Hemmer]

We're seeing an issue on our systems where syslog-ng is consuming an
excessive amount of memory, approximately 380mb (RSS) right now on one
of our hosts.
I'm not sure what's causing this as our config is pretty minimal, and
there are no destinations unavailable which would cause it to buffer. We
do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might
have caused it to buffer a large amount some time in the past, but would
it not release these buffers?

Version 3.13.2 on CentOS/7 (RPM from
https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/)

Config:
@version:3.13
@include "scl.conf"

# This configuration is meant to utilize systemd journald
options {
  flush_lines(0);
  time_reopen(10);
  log-msg-size(65536);
  log_fifo_size(10000);
  chain_hostnames(off);
  use_dns(no);
  use_fqdn(yes);
  keep_hostname(yes);
  create_dirs(no);
  stats_freq(3600);
};

source s_system { system(); };
source s_internal { internal(); };

destination d_journald {
  unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n"));
};

log { source(s_internal); destination(d_journald); };

destination d_loghost {
    syslog("syslog.ecom.chewy.com"
        frac-digits(3)
        log-fifo-size(10000)
    );
};

log { source(s_system); destination(d_loghost); };

Stats:

# syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1
dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216
destination;d_journald;;a;processed;1997
src.internal;s_internal#0;;a;processed;1997
src.internal;s_internal#0;;a;stamp;1515709115
center;;received;a;processed;1997
destination;d_loghost;;a;processed;489030242
source;s_internal;;a;processed;1997
center;;queued;a;processed;489032239
global;scratch_buffers_count;;a;queued;25769803782
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0
dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845
global;payload_reallocs;;a;processed;2932894697
src.journald;;journal;a;processed;489030242
src.journald;;journal;a;stamp;1515710081
global;sdata_updates;;a;processed;0
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;135
global;internal_queue_length;;a;processed;0

Thanks

-Patrick


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180111/8f3cef70/attachment.html>