[syslog-ng] Excessive memory usage

[Scheidler, Balázs]

Thanks. It still might make sense to try the jemalloc allocator if memory
pressure is an issue. Afaik it is always used in our syslog-ng store box
product, for better performance and more efficient memory handling.

On Jan 25, 2018 22:47, "Patrick Hemmer" <syslogng at stormcloud9.net> wrote:

> Just to follow up on this, it does appear that the large buffers were the
> issue. We switched to using disk buffering instead, and the memory usage is
> much better now. The app starts out at 8mb used, goes up to 20mb over a few
> seconds, then drops back down to 8mb.
>
> -Patrick
>
> On 2018/1/16 08:00, Scheidler, Balázs wrote:
>
> Hi,
>
> we do release those buffers, but the libc malloc() may not.
>
> It could make sense to LD_PRELOAD the jemalloc allocator, that is better
> at returning pages to the system.
>
>
> --
> Bazsi
>
> On Thu, Jan 11, 2018 at 11:54 PM, Patrick Hemmer <syslogng at stormcloud9.net
> > wrote:
>
>> We're seeing an issue on our systems where syslog-ng is consuming an
>> excessive amount of memory, approximately 380mb (RSS) right now on one of
>> our hosts.
>> I'm not sure what's causing this as our config is pretty minimal, and
>> there are no destinations unavailable which would cause it to buffer. We do
>> have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have
>> caused it to buffer a large amount some time in the past, but would it not
>> release these buffers?
>>
>> Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud.
>> org/coprs/czanik/syslog-ng313/)
>>
>> Config:
>> @version:3.13
>> @include "scl.conf"
>>
>> # This configuration is meant to utilize systemd journald
>> options {
>>   flush_lines(0);
>>   time_reopen(10);
>>   log-msg-size(65536);
>>   log_fifo_size(10000);
>>   chain_hostnames(off);
>>   use_dns(no);
>>   use_fqdn(yes);
>>   keep_hostname(yes);
>>   create_dirs(no);
>>   stats_freq(3600);
>> };
>>
>> source s_system { system(); };
>> source s_internal { internal(); };
>>
>> destination d_journald {
>>   unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n"));
>> };
>>
>> log { source(s_internal); destination(d_journald); };
>>
>> destination d_loghost {
>>     syslog("syslog.ecom.chewy.com"
>>         frac-digits(3)
>>         log-fifo-size(10000)
>>     );
>> };
>>
>> log { source(s_system); destination(d_loghost); };
>>
>> Stats:
>>
>> # syslog-ng-ctl stats
>> SourceName;SourceId;SourceInstance;State;Type;Number
>> dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025
>> dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;proce
>> ssed;489030242
>> dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1
>> dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216
>> destination;d_journald;;a;processed;1997
>> src.internal;s_internal#0;;a;processed;1997
>> src.internal;s_internal#0;;a;stamp;1515709115
>> center;;received;a;processed;1997
>> destination;d_loghost;;a;processed;489030242
>> source;s_internal;;a;processed;1997
>> center;;queued;a;processed;489032239
>> global;scratch_buffers_count;;a;queued;25769803782
>> dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev
>> /log;a;dropped;0
>> dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev
>> /log;a;processed;845
>> dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev
>> /log;a;queued;0
>> dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev
>> /log;a;written;845
>> global;payload_reallocs;;a;processed;2932894697
>> src.journald;;journal;a;processed;489030242
>> src.journald;;journal;a;stamp;1515710081
>> global;sdata_updates;;a;processed;0
>> global;scratch_buffers_bytes;;a;queued;0
>> global;msg_clones;;a;processed;135
>> global;internal_queue_length;;a;processed;0
>>
>> Thanks
>>
>> -Patrick
>>
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180126/18a7c573/attachment.html>