[syslog-ng] syslog-ng 3.22.1 consuming a lot of RAM on amqp or kafka destinations - patterndb

[joao reis]

Ok, I did another tests and discover that the problem is not amqp or 
kafka destination, but the parserdb.

Removing the parser in the log line, the memory keeps normal.

Downgrading to syslog-ng 3.17 I can use the parserdb with amqp without 
the memory leak problem. I didn't test with newer versions yet, but 
seems to be a memory leak problem with patterndb. I will remove the 
"<action>" sections in patterndb to see if it change something in 3.22...

Any known issue?

On 8/24/19 9:18 AM, joao reis wrote:
> Hi,
>
> I'm having high RAM consuming problems using both amqp or kafka 
> destinations. I'm using patterndb (without any context definition) 
> some basic filters.
>
> Changing the destination to "file" or "program" type, the high RAM 
> problem doesn't happen.
>
> I tested these backends only in this version of syslog-ng and I don't 
> know if it is a version specific problem.
>
> The backends are up and running and receiving correctly from syslog-ng.
>
> Here are my configurations:
>
> destination d_amqp {
>     amqp(
>         vhost("/email_log")
>         host("127.0.0.1")
>         port(5672)
>         exchange("maillog")
>         username("maillog")
>         password("maillog")
>         disk-buffer(
>             mem-buf-size(10000)
>             disk-buf-size(2000000)
>             reliable(yes)
>             dir("/tmp/disk-buffer")
>         )
>         exchange-type("fanout")
>         routing-key("")
>         persistent(yes)
>         value-pairs(
>             scope("nv-pairs")
>         )
>     );
> };
>
>
> destination d_kafka {
>   kafka(
>     bootstrap-servers("127.0.0.1:9092")
>     topic("MYTOPIC")
>   );
> };
>
>
> I added the disk-buffer option to see if it gets any better, but 
> without any good result;
>
> Here the syslog-ng-ctl output, when configured to amqp:
>
> [root at logserver3 conf.d]# syslog-ng-ctl stats
> SourceName;SourceId;SourceInstance;State;Type;Number
> global;payload_reallocs;;a;processed;3635299
> src.journald;s_sys#0;journal;a;processed;542
> src.journald;s_sys#0;journal;a;stamp;1566648214
> global;sdata_updates;;a;processed;0
> global;scratch_buffers_bytes;;a;queued;512
> source;s_host;;a;processed;5440633
> destination;d_boot;;a;processed;0
> destination;d_kern;;a;processed;9
> source;s_sys;;a;processed;612
> dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0 
>
> dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676 
>
> dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0 
>
> dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676 
>
> src.internal;s_sys#1;;a;processed;70
> src.internal;s_sys#1;;a;stamp;1566648339
> global;msg_clones;;a;processed;2691634
> global;internal_queue_length;;a;processed;0
> destination;d_spol;;a;processed;0
> destination;d_mlal;;a;processed;0
> center;;received;a;processed;5441245
> destination;d_mesg;;a;processed;531
> destination;d_mail;;a;processed;0
> destination;d_auth;;a;processed;29
> destination;d_cron;;a;processed;52
> destination;d_amqp;;a;processed;2691676
> global;scratch_buffers_count;;a;queued;5970004541496
> center;;queued;a;processed;2692297
>
> Thanks in advance.
>
>
>
> ______________________________________________________________________________ 
>
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: 
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>