[syslog-ng] syslog-ng 3.22.1 consuming a lot of RAM on amqp or kafka destinations

[joao reis]

Hi,

I'm having high RAM consuming problems using both amqp or kafka 
destinations. I'm using patterndb (without any context definition) some 
basic filters.

Changing the destination to "file" or "program" type, the high RAM 
problem doesn't happen.

I tested these backends only in this version of syslog-ng and I don't 
know if it is a version specific problem.

The backends are up and running and receiving correctly from syslog-ng.

Here are my configurations:

destination d_amqp {
     amqp(
         vhost("/email_log")
         host("127.0.0.1")
         port(5672)
         exchange("maillog")
         username("maillog")
         password("maillog")
         disk-buffer(
             mem-buf-size(10000)
             disk-buf-size(2000000)
             reliable(yes)
             dir("/tmp/disk-buffer")
         )
         exchange-type("fanout")
         routing-key("")
         persistent(yes)
         value-pairs(
             scope("nv-pairs")
         )
     );
};


destination d_kafka {
   kafka(
     bootstrap-servers("127.0.0.1:9092")
     topic("MYTOPIC")
   );
};


I added the disk-buffer option to see if it gets any better, but without 
any good result;

Here the syslog-ng-ctl output, when configured to amqp:

[root at logserver3 conf.d]# syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
global;payload_reallocs;;a;processed;3635299
src.journald;s_sys#0;journal;a;processed;542
src.journald;s_sys#0;journal;a;stamp;1566648214
global;sdata_updates;;a;processed;0
global;scratch_buffers_bytes;;a;queued;512
source;s_host;;a;processed;5440633
destination;d_boot;;a;processed;0
destination;d_kern;;a;processed;9
source;s_sys;;a;processed;612
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676
src.internal;s_sys#1;;a;processed;70
src.internal;s_sys#1;;a;stamp;1566648339
global;msg_clones;;a;processed;2691634
global;internal_queue_length;;a;processed;0
destination;d_spol;;a;processed;0
destination;d_mlal;;a;processed;0
center;;received;a;processed;5441245
destination;d_mesg;;a;processed;531
destination;d_mail;;a;processed;0
destination;d_auth;;a;processed;29
destination;d_cron;;a;processed;52
destination;d_amqp;;a;processed;2691676
global;scratch_buffers_count;;a;queued;5970004541496
center;;queued;a;processed;2692297

Thanks in advance.