syslog-ng 3.22.1 consuming a lot of RAM on amqp or kafka destinations
Hi, I'm having high RAM consuming problems using both amqp or kafka destinations. I'm using patterndb (without any context definition) some basic filters. Changing the destination to "file" or "program" type, the high RAM problem doesn't happen. I tested these backends only in this version of syslog-ng and I don't know if it is a version specific problem. The backends are up and running and receiving correctly from syslog-ng. Here are my configurations: destination d_amqp { amqp( vhost("/email_log") host("127.0.0.1") port(5672) exchange("maillog") username("maillog") password("maillog") disk-buffer( mem-buf-size(10000) disk-buf-size(2000000) reliable(yes) dir("/tmp/disk-buffer") ) exchange-type("fanout") routing-key("") persistent(yes) value-pairs( scope("nv-pairs") ) ); }; destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("MYTOPIC") ); }; I added the disk-buffer option to see if it gets any better, but without any good result; Here the syslog-ng-ctl output, when configured to amqp: [root@logserver3 conf.d]# syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number global;payload_reallocs;;a;processed;3635299 src.journald;s_sys#0;journal;a;processed;542 src.journald;s_sys#0;journal;a;stamp;1566648214 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;512 source;s_host;;a;processed;5440633 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;9 source;s_sys;;a;processed;612 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676 src.internal;s_sys#1;;a;processed;70 src.internal;s_sys#1;;a;stamp;1566648339 global;msg_clones;;a;processed;2691634 global;internal_queue_length;;a;processed;0 destination;d_spol;;a;processed;0 destination;d_mlal;;a;processed;0 center;;received;a;processed;5441245 destination;d_mesg;;a;processed;531 destination;d_mail;;a;processed;0 destination;d_auth;;a;processed;29 destination;d_cron;;a;processed;52 destination;d_amqp;;a;processed;2691676 global;scratch_buffers_count;;a;queued;5970004541496 center;;queued;a;processed;2692297 Thanks in advance.
Ok, I did another tests and discover that the problem is not amqp or kafka destination, but the parserdb. Removing the parser in the log line, the memory keeps normal. Downgrading to syslog-ng 3.17 I can use the parserdb with amqp without the memory leak problem. I didn't test with newer versions yet, but seems to be a memory leak problem with patterndb. I will remove the "<action>" sections in patterndb to see if it change something in 3.22... Any known issue? On 8/24/19 9:18 AM, joao reis wrote:
Hi,
I'm having high RAM consuming problems using both amqp or kafka destinations. I'm using patterndb (without any context definition) some basic filters.
Changing the destination to "file" or "program" type, the high RAM problem doesn't happen.
I tested these backends only in this version of syslog-ng and I don't know if it is a version specific problem.
The backends are up and running and receiving correctly from syslog-ng.
Here are my configurations:
destination d_amqp { amqp( vhost("/email_log") host("127.0.0.1") port(5672) exchange("maillog") username("maillog") password("maillog") disk-buffer( mem-buf-size(10000) disk-buf-size(2000000) reliable(yes) dir("/tmp/disk-buffer") ) exchange-type("fanout") routing-key("") persistent(yes) value-pairs( scope("nv-pairs") ) ); };
destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("MYTOPIC") ); };
I added the disk-buffer option to see if it gets any better, but without any good result;
Here the syslog-ng-ctl output, when configured to amqp:
[root@logserver3 conf.d]# syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number global;payload_reallocs;;a;processed;3635299 src.journald;s_sys#0;journal;a;processed;542 src.journald;s_sys#0;journal;a;stamp;1566648214 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;512 source;s_host;;a;processed;5440633 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;9 source;s_sys;;a;processed;612 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676
src.internal;s_sys#1;;a;processed;70 src.internal;s_sys#1;;a;stamp;1566648339 global;msg_clones;;a;processed;2691634 global;internal_queue_length;;a;processed;0 destination;d_spol;;a;processed;0 destination;d_mlal;;a;processed;0 center;;received;a;processed;5441245 destination;d_mesg;;a;processed;531 destination;d_mail;;a;processed;0 destination;d_auth;;a;processed;29 destination;d_cron;;a;processed;52 destination;d_amqp;;a;processed;2691676 global;scratch_buffers_count;;a;queued;5970004541496 center;;queued;a;processed;2692297
Thanks in advance.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Joao, Could you please share the whole config file, or the minimal config parts for the problem to happen, so we can reproduce the issue locally? The patterndb xml file will probably also be needed. There were some PRs merged related to patterndb since 3.17, but I cannot say for sure, which would introduce a memory leak. If you would like to debug syslog-ng for yourself, you can run it with valgrind. For example: valgrind --leak-check=full sbin/syslog-ng -Fedtv Thanks! Best regards, Attila ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of joao reis <joao@7lan.net> Sent: Sunday, August 25, 2019 9:19 PM To: syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng 3.22.1 consuming a lot of RAM on amqp or kafka destinations - patterndb CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Ok, I did another tests and discover that the problem is not amqp or kafka destination, but the parserdb. Removing the parser in the log line, the memory keeps normal. Downgrading to syslog-ng 3.17 I can use the parserdb with amqp without the memory leak problem. I didn't test with newer versions yet, but seems to be a memory leak problem with patterndb. I will remove the "<action>" sections in patterndb to see if it change something in 3.22... Any known issue? On 8/24/19 9:18 AM, joao reis wrote:
Hi,
I'm having high RAM consuming problems using both amqp or kafka destinations. I'm using patterndb (without any context definition) some basic filters.
Changing the destination to "file" or "program" type, the high RAM problem doesn't happen.
I tested these backends only in this version of syslog-ng and I don't know if it is a version specific problem.
The backends are up and running and receiving correctly from syslog-ng.
Here are my configurations:
destination d_amqp { amqp( vhost("/email_log") host("127.0.0.1") port(5672) exchange("maillog") username("maillog") password("maillog") disk-buffer( mem-buf-size(10000) disk-buf-size(2000000) reliable(yes) dir("/tmp/disk-buffer") ) exchange-type("fanout") routing-key("") persistent(yes) value-pairs( scope("nv-pairs") ) ); };
destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("MYTOPIC") ); };
I added the disk-buffer option to see if it gets any better, but without any good result;
Here the syslog-ng-ctl output, when configured to amqp:
[root@logserver3 conf.d]# syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number global;payload_reallocs;;a;processed;3635299 src.journald;s_sys#0;journal;a;processed;542 src.journald;s_sys#0;journal;a;stamp;1566648214 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;512 source;s_host;;a;processed;5440633 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;9 source;s_sys;;a;processed;612 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676
src.internal;s_sys#1;;a;processed;70 src.internal;s_sys#1;;a;stamp;1566648339 global;msg_clones;;a;processed;2691634 global;internal_queue_length;;a;processed;0 destination;d_spol;;a;processed;0 destination;d_mlal;;a;processed;0 center;;received;a;processed;5441245 destination;d_mesg;;a;processed;531 destination;d_mail;;a;processed;0 destination;d_auth;;a;processed;29 destination;d_cron;;a;processed;52 destination;d_amqp;;a;processed;2691676 global;scratch_buffers_count;;a;queued;5970004541496 center;;queued;a;processed;2692297
Thanks in advance.
______________________________________________________________________________
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=IevvrtaCVev4wWIIkX2qQ3USXFqgswZE%2Fr%2BSN7fps7I%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=o4nPzuf08QzuA0gWM2YhR46tBfzYxCk2rEcAHB%2B2BYM%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=1UinGzeqP9y8I42N0m2pnTtSYk%2BCzpN7SWuhPhGyhLs%3D&reserved=0
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=IevvrtaCVev4wWIIkX2qQ3USXFqgswZE%2Fr%2BSN7fps7I%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=o4nPzuf08QzuA0gWM2YhR46tBfzYxCk2rEcAHB%2B2BYM%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=1UinGzeqP9y8I42N0m2pnTtSYk%2BCzpN7SWuhPhGyhLs%3D&reserved=0
Hi, Sorry for the delay, I've attached the configuration files. The problem occurs in 3.24 too. I will run valgrind to try to find out something. Thanks in advance. On 8/26/19 4:32 AM, Attila Szakacs (aszakacs) wrote:
Hi Joao,
Could you please share the whole config file, or the minimal config parts for the problem to happen, so we can reproduce the issue locally? The patterndb xml file will probably also be needed. There were some PRs merged related to patterndb since 3.17, but I cannot say for sure, which would introduce a memory leak.
If you would like to debug syslog-ng for yourself, you can run it with valgrind. For example: valgrind --leak-check=full sbin/syslog-ng -Fedtv
Thanks!
Best regards, Attila ------------------------------------------------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of joao reis <joao@7lan.net> *Sent:* Sunday, August 25, 2019 9:19 PM *To:* syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] syslog-ng 3.22.1 consuming a lot of RAM on amqp or kafka destinations - patterndb CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Ok, I did another tests and discover that the problem is not amqp or kafka destination, but the parserdb.
Removing the parser in the log line, the memory keeps normal.
Downgrading to syslog-ng 3.17 I can use the parserdb with amqp without the memory leak problem. I didn't test with newer versions yet, but seems to be a memory leak problem with patterndb. I will remove the "<action>" sections in patterndb to see if it change something in 3.22...
Any known issue?
On 8/24/19 9:18 AM, joao reis wrote:
Hi,
I'm having high RAM consuming problems using both amqp or kafka destinations. I'm using patterndb (without any context definition) some basic filters.
Changing the destination to "file" or "program" type, the high RAM problem doesn't happen.
I tested these backends only in this version of syslog-ng and I don't know if it is a version specific problem.
The backends are up and running and receiving correctly from syslog-ng.
Here are my configurations:
destination d_amqp { amqp( vhost("/email_log") host("127.0.0.1") port(5672) exchange("maillog") username("maillog") password("maillog") disk-buffer( mem-buf-size(10000) disk-buf-size(2000000) reliable(yes) dir("/tmp/disk-buffer") ) exchange-type("fanout") routing-key("") persistent(yes) value-pairs( scope("nv-pairs") ) ); };
destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("MYTOPIC") ); };
I added the disk-buffer option to see if it gets any better, but without any good result;
Here the syslog-ng-ctl output, when configured to amqp:
[root@logserver3 conf.d]# syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number global;payload_reallocs;;a;processed;3635299 src.journald;s_sys#0;journal;a;processed;542 src.journald;s_sys#0;journal;a;stamp;1566648214 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;512 source;s_host;;a;processed;5440633 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;9 source;s_sys;;a;processed;612
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676
src.internal;s_sys#1;;a;processed;70 src.internal;s_sys#1;;a;stamp;1566648339 global;msg_clones;;a;processed;2691634 global;internal_queue_length;;a;processed;0 destination;d_spol;;a;processed;0 destination;d_mlal;;a;processed;0 center;;received;a;processed;5441245 destination;d_mesg;;a;processed;531 destination;d_mail;;a;processed;0 destination;d_auth;;a;processed;29 destination;d_cron;;a;processed;52 destination;d_amqp;;a;processed;2691676 global;scratch_buffers_count;;a;queued;5970004541496 center;;queued;a;processed;2692297
Thanks in advance.
______________________________________________________________________________
Member info:
Documentation:
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=IevvrtaCVev4wWIIkX2qQ3USXFqgswZE%2Fr%2BSN7fps7I%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=o4nPzuf08QzuA0gWM2YhR46tBfzYxCk2rEcAHB%2B2BYM%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=1UinGzeqP9y8I42N0m2pnTtSYk%2BCzpN7SWuhPhGyhLs%3D&reserved=0
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Attila Szakacs (aszakacs)
-
joao reis