Thanks. It still might make sense to try the jemalloc allocator if memory pressure is an issue. Afaik it is always used in our syslog-ng store box product, for better performance and more efficient memory handling. On Jan 25, 2018 22:47, "Patrick Hemmer" <syslogng@stormcloud9.net> wrote:
Just to follow up on this, it does appear that the large buffers were the issue. We switched to using disk buffering instead, and the memory usage is much better now. The app starts out at 8mb used, goes up to 20mb over a few seconds, then drops back down to 8mb.
-Patrick
On 2018/1/16 08:00, Scheidler, Balázs wrote:
Hi,
we do release those buffers, but the libc malloc() may not.
It could make sense to LD_PRELOAD the jemalloc allocator, that is better at returning pages to the system.
-- Bazsi
On Thu, Jan 11, 2018 at 11:54 PM, Patrick Hemmer <syslogng@stormcloud9.net
wrote:
We're seeing an issue on our systems where syslog-ng is consuming an excessive amount of memory, approximately 380mb (RSS) right now on one of our hosts. I'm not sure what's causing this as our config is pretty minimal, and there are no destinations unavailable which would cause it to buffer. We do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have caused it to buffer a large amount some time in the past, but would it not release these buffers?
Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud. org/coprs/czanik/syslog-ng313/)
Config: @version:3.13 @include "scl.conf"
# This configuration is meant to utilize systemd journald options { flush_lines(0); time_reopen(10); log-msg-size(65536); log_fifo_size(10000); chain_hostnames(off); use_dns(no); use_fqdn(yes); keep_hostname(yes); create_dirs(no); stats_freq(3600); };
source s_system { system(); }; source s_internal { internal(); };
destination d_journald { unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n")); };
log { source(s_internal); destination(d_journald); };
destination d_loghost { syslog("syslog.ecom.chewy.com" frac-digits(3) log-fifo-size(10000) ); };
log { source(s_system); destination(d_loghost); };
Stats:
# syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;proce ssed;489030242 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216 destination;d_journald;;a;processed;1997 src.internal;s_internal#0;;a;processed;1997 src.internal;s_internal#0;;a;stamp;1515709115 center;;received;a;processed;1997 destination;d_loghost;;a;processed;489030242 source;s_internal;;a;processed;1997 center;;queued;a;processed;489032239 global;scratch_buffers_count;;a;queued;25769803782 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev /log;a;dropped;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev /log;a;processed;845 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev /log;a;queued;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev /log;a;written;845 global;payload_reallocs;;a;processed;2932894697 src.journald;;journal;a;processed;489030242 src.journald;;journal;a;stamp;1515710081 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;135 global;internal_queue_length;;a;processed;0
Thanks
-Patrick
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq