<div dir="auto">Thanks. It still might make sense to try the jemalloc allocator if memory pressure is an issue. Afaik it is always used in our syslog-ng store box product, for better performance and more efficient memory handling.</div><div class="gmail_extra"><br><div class="gmail_quote">On Jan 25, 2018 22:47, "Patrick Hemmer" <<a href="mailto:syslogng@stormcloud9.net">syslogng@stormcloud9.net</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Just to follow up on this, it does appear that the large buffers
were the issue. We switched to using disk buffering instead, and the
memory usage is much better now. The app starts out at 8mb used,
goes up to 20mb over a few seconds, then drops back down to 8mb.<br>
<br>
-Patrick<br>
<br>
<div class="m_3411098051013120054moz-cite-prefix">On 2018/1/16 08:00, Scheidler, Balázs
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi,<br>
<br>
</div>
we do release those buffers, but the libc malloc() may not. <br>
<br>
It could make sense to LD_PRELOAD the jemalloc allocator, that
is better at returning pages to the system.<br>
<br>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_3411098051013120054gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">-- <br>
Bazsi<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Thu, Jan 11, 2018 at 11:54 PM,
Patrick Hemmer <span dir="ltr"><<a href="mailto:syslogng@stormcloud9.net" target="_blank">syslogng@stormcloud9.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> We're seeing an issue
on our systems where syslog-ng is consuming an excessive
amount of memory, approximately 380mb (RSS) right now on
one of our hosts.<br>
I'm not sure what's causing this as our config is pretty
minimal, and there are no destinations unavailable which
would cause it to buffer. We do have `log-msg-size(65536)`
and `log-fifo-size(10000)` which might have caused it to
buffer a large amount some time in the past, but would it
not release these buffers?<br>
<br>
Version 3.13.2 on CentOS/7 (RPM from <a class="m_3411098051013120054m_-1350886579924916691moz-txt-link-freetext" href="https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/" target="_blank">https://copr.fedorainfracloud.<wbr>org/coprs/czanik/syslog-ng313/</a><wbr>)<br>
<br>
Config:<br>
<font color="#990000"><tt>@version:3.13</tt><tt><br>
</tt><tt>@include "scl.conf"</tt><tt><br>
</tt><tt><br>
</tt><tt># This configuration is meant to utilize
systemd journald</tt><tt><br>
</tt><tt>options {</tt><tt><br>
</tt><tt> flush_lines(0);</tt><tt><br>
</tt><tt> time_reopen(10);</tt><tt><br>
</tt><tt> log-msg-size(65536);</tt><tt><br>
</tt><tt> log_fifo_size(10000);</tt><tt><br>
</tt><tt> chain_hostnames(off);</tt><tt><br>
</tt><tt> use_dns(no);</tt><tt><br>
</tt><tt> use_fqdn(yes);</tt><tt><br>
</tt><tt> keep_hostname(yes);</tt><tt><br>
</tt><tt> create_dirs(no);</tt><tt><br>
</tt><tt> stats_freq(3600);</tt><tt><br>
</tt><tt>};</tt><tt><br>
</tt><tt><br>
</tt><tt>source s_system { system(); };</tt><tt><br>
</tt><tt>source s_internal { internal(); };</tt><tt><br>
</tt><tt><br>
</tt><tt>destination d_journald {</tt><tt><br>
</tt><tt> unix-dgram("/dev/log" flush-lines(1)
template("${MSG}\n"));</tt><tt><br>
</tt><tt>};</tt><tt><br>
</tt><tt><br>
</tt><tt>log { source(s_internal);
destination(d_journald); };</tt><tt><br>
</tt><tt><br>
</tt><tt>destination d_loghost {</tt><tt><br>
</tt><tt> syslog("<a href="http://syslog.ecom.chewy.com" target="_blank">syslog.ecom.chewy.com</a>"</tt><tt><br>
</tt><tt> frac-digits(3)</tt><tt><br>
</tt><tt> log-fifo-size(10000)</tt><tt><br>
</tt><tt> );</tt><tt><br>
</tt><tt>};</tt><tt><br>
</tt><tt><br>
</tt><tt>log { source(s_system); destination(d_loghost);
};</tt></font><br>
<br>
Stats:<br>
<br>
<font color="#990000"><tt># syslog-ng-ctl stats</tt><tt><br>
</tt><tt>SourceName;SourceId;SourceInst<wbr>ance;State;Type;Number</tt><tt><br>
</tt><tt>dst.syslog;d_loghost#0;tcp,sys<wbr>log.ecom.chewy.com:601;a;dropp<wbr>ed;163025</tt><tt><br>
</tt><tt>dst.syslog;d_loghost#0;tcp,sys<wbr>log.ecom.chewy.com:601;a;proce<wbr>ssed;489030242</tt><tt><br>
</tt><tt>dst.syslog;d_loghost#0;tcp,sys<wbr>log.ecom.chewy.com:601;a;queue<wbr>d;1</tt><tt><br>
</tt><tt>dst.syslog;d_loghost#0;tcp,sys<wbr>log.ecom.chewy.com:601;a;writt<wbr>en;488867216</tt><tt><br>
</tt><tt>destination;d_journald;;a;proc<wbr>essed;1997</tt><tt><br>
</tt><tt>src.internal;s_internal#0;;a;p<wbr>rocessed;1997</tt><tt><br>
</tt><tt>src.internal;s_internal#0;;a;s<wbr>tamp;1515709115</tt><tt><br>
</tt><tt>center;;received;a;processed;1<wbr>997</tt><tt><br>
</tt><tt>destination;d_loghost;;a;proce<wbr>ssed;489030242</tt><tt><br>
</tt><tt>source;s_internal;;a;processed<wbr>;1997</tt><tt><br>
</tt><tt>center;;queued;a;processed;489<wbr>032239</tt><tt><br>
</tt><tt>global;scratch_buffers_count;;<wbr>a;queued;25769803782</tt><tt><br>
</tt><tt>dst.unix-dgram;d_journald#0;un<wbr>ix-dgram,localhost.afunix:/dev<wbr>/log;a;dropped;0</tt><tt><br>
</tt><tt>dst.unix-dgram;d_journald#0;un<wbr>ix-dgram,localhost.afunix:/dev<wbr>/log;a;processed;845</tt><tt><br>
</tt><tt>dst.unix-dgram;d_journald#0;un<wbr>ix-dgram,localhost.afunix:/dev<wbr>/log;a;queued;0</tt><tt><br>
</tt><tt>dst.unix-dgram;d_journald#0;un<wbr>ix-dgram,localhost.afunix:/dev<wbr>/log;a;written;845</tt><tt><br>
</tt><tt>global;payload_reallocs;;a;pro<wbr>cessed;2932894697</tt><tt><br>
</tt><tt>src.journald;;journal;a;proces<wbr>sed;489030242</tt><tt><br>
</tt><tt>src.journald;;journal;a;stamp;<wbr>1515710081</tt><tt><br>
</tt><tt>global;sdata_updates;;a;proces<wbr>sed;0</tt><tt><br>
</tt><tt>global;scratch_buffers_bytes;;<wbr>a;queued;0</tt><tt><br>
</tt><tt>global;msg_clones;;a;processed<wbr>;135</tt><tt><br>
</tt><tt>global;internal_queue_length;;<wbr>a;processed;0</tt></font><br>
<br>
Thanks<span class="m_3411098051013120054HOEnZb"><font color="#888888"><br>
<br>
-Patrick<br>
<br>
<br>
</font></span></div>
<br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_3411098051013120054mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>______________________________<wbr>__________________
Member info: <a class="m_3411098051013120054moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a>
Documentation: <a class="m_3411098051013120054moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a>
FAQ: <a class="m_3411098051013120054moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</div>
</blockquote></div></div>