<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Hi,</p>
<p><br>
</p>
<p>Sorry for the delay, I've attached the configuration files. <br>
</p>
<p>The problem occurs in 3.24 too.</p>
<p>I will run valgrind to try to find out something.</p>
<p><br>
</p>
<p>Thanks in advance.<br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 8/26/19 4:32 AM, Attila Szakacs
(aszakacs) wrote:<br>
</div>
<blockquote type="cite"
cite="mid:MN2PR19MB322974486CE3BA8A3D5D9B109EA10@MN2PR19MB3229.namprd19.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Hi Joao,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Could you please share the whole config file, or the minimal
config parts for the problem to happen, so we can reproduce the
issue locally? The patterndb xml file will probably also be
needed.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
There were some PRs merged related to patterndb since 3.17, but
I cannot say for sure, which would introduce a memory leak.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
If you would like to debug syslog-ng for yourself, you can run
it with valgrind.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
For example: valgrind --leak-check=full sbin/syslog-ng -Fedtv</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Thanks!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Attila</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
syslog-ng <a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng-bounces@lists.balabit.hu"><syslog-ng-bounces@lists.balabit.hu></a> on behalf
of joao reis <a class="moz-txt-link-rfc2396E" href="mailto:joao@7lan.net"><joao@7lan.net></a><br>
<b>Sent:</b> Sunday, August 25, 2019 9:19 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>
<a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng@lists.balabit.hu"><syslog-ng@lists.balabit.hu></a><br>
<b>Subject:</b> Re: [syslog-ng] syslog-ng 3.22.1 consuming a
lot of RAM on amqp or kafka destinations - patterndb</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span
style="font-size:11pt;">
<div class="PlainText">CAUTION: This email originated from
outside of the organization. Do not follow guidance, click
links, or open attachments unless you recognize the sender
and know the content is safe.<br>
<br>
<br>
Ok, I did another tests and discover that the problem is
not amqp or<br>
kafka destination, but the parserdb.<br>
<br>
Removing the parser in the log line, the memory keeps
normal.<br>
<br>
Downgrading to syslog-ng 3.17 I can use the parserdb with
amqp without<br>
the memory leak problem. I didn't test with newer versions
yet, but<br>
seems to be a memory leak problem with patterndb. I will
remove the<br>
"<action>" sections in patterndb to see if it change
something in 3.22...<br>
<br>
Any known issue?<br>
<br>
On 8/24/19 9:18 AM, joao reis wrote:<br>
> Hi,<br>
><br>
> I'm having high RAM consuming problems using both
amqp or kafka<br>
> destinations. I'm using patterndb (without any
context definition)<br>
> some basic filters.<br>
><br>
> Changing the destination to "file" or "program" type,
the high RAM<br>
> problem doesn't happen.<br>
><br>
> I tested these backends only in this version of
syslog-ng and I don't<br>
> know if it is a version specific problem.<br>
><br>
> The backends are up and running and receiving
correctly from syslog-ng.<br>
><br>
> Here are my configurations:<br>
><br>
> destination d_amqp {<br>
> amqp(<br>
> vhost("/email_log")<br>
> host("127.0.0.1")<br>
> port(5672)<br>
> exchange("maillog")<br>
> username("maillog")<br>
> password("maillog")<br>
> disk-buffer(<br>
> mem-buf-size(10000)<br>
> disk-buf-size(2000000)<br>
> reliable(yes)<br>
> dir("/tmp/disk-buffer")<br>
> )<br>
> exchange-type("fanout")<br>
> routing-key("")<br>
> persistent(yes)<br>
> value-pairs(<br>
> scope("nv-pairs")<br>
> )<br>
> );<br>
> };<br>
><br>
><br>
> destination d_kafka {<br>
> kafka(<br>
> bootstrap-servers("127.0.0.1:9092")<br>
> topic("MYTOPIC")<br>
> );<br>
> };<br>
><br>
><br>
> I added the disk-buffer option to see if it gets any
better, but<br>
> without any good result;<br>
><br>
> Here the syslog-ng-ctl output, when configured to
amqp:<br>
><br>
> [root@logserver3 conf.d]# syslog-ng-ctl stats<br>
> SourceName;SourceId;SourceInstance;State;Type;Number<br>
> global;payload_reallocs;;a;processed;3635299<br>
> src.journald;s_sys#0;journal;a;processed;542<br>
> src.journald;s_sys#0;journal;a;stamp;1566648214<br>
> global;sdata_updates;;a;processed;0<br>
> global;scratch_buffers_bytes;;a;queued;512<br>
> source;s_host;;a;processed;5440633<br>
> destination;d_boot;;a;processed;0<br>
> destination;d_kern;;a;processed;9<br>
> source;s_sys;;a;processed;612<br>
>
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0<br>
><br>
>
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676<br>
><br>
>
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0<br>
><br>
>
dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676<br>
><br>
> src.internal;s_sys#1;;a;processed;70<br>
> src.internal;s_sys#1;;a;stamp;1566648339<br>
> global;msg_clones;;a;processed;2691634<br>
> global;internal_queue_length;;a;processed;0<br>
> destination;d_spol;;a;processed;0<br>
> destination;d_mlal;;a;processed;0<br>
> center;;received;a;processed;5441245<br>
> destination;d_mesg;;a;processed;531<br>
> destination;d_mail;;a;processed;0<br>
> destination;d_auth;;a;processed;29<br>
> destination;d_cron;;a;processed;52<br>
> destination;d_amqp;;a;processed;2691676<br>
> global;scratch_buffers_count;;a;queued;5970004541496<br>
> center;;queued;a;processed;2692297<br>
><br>
> Thanks in advance.<br>
><br>
><br>
><br>
>
______________________________________________________________________________<br>
><br>
> Member info: <a
href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=IevvrtaCVev4wWIIkX2qQ3USXFqgswZE%2Fr%2BSN7fps7I%3D&reserved=0"
moz-do-not-send="true">
https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=IevvrtaCVev4wWIIkX2qQ3USXFqgswZE%2Fr%2BSN7fps7I%3D&reserved=0</a><br>
> Documentation:<br>
> <a
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=o4nPzuf08QzuA0gWM2YhR46tBfzYxCk2rEcAHB%2B2BYM%3D&reserved=0"
moz-do-not-send="true">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=o4nPzuf08QzuA0gWM2YhR46tBfzYxCk2rEcAHB%2B2BYM%3D&reserved=0</a><br>
> FAQ: <a
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=1UinGzeqP9y8I42N0m2pnTtSYk%2BCzpN7SWuhPhGyhLs%3D&reserved=0"
moz-do-not-send="true">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=1UinGzeqP9y8I42N0m2pnTtSYk%2BCzpN7SWuhPhGyhLs%3D&reserved=0</a><br>
><br>
______________________________________________________________________________<br>
Member info: <a
href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=IevvrtaCVev4wWIIkX2qQ3USXFqgswZE%2Fr%2BSN7fps7I%3D&reserved=0"
moz-do-not-send="true">
https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=IevvrtaCVev4wWIIkX2qQ3USXFqgswZE%2Fr%2BSN7fps7I%3D&reserved=0</a><br>
Documentation: <a
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=o4nPzuf08QzuA0gWM2YhR46tBfzYxCk2rEcAHB%2B2BYM%3D&reserved=0"
moz-do-not-send="true">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=o4nPzuf08QzuA0gWM2YhR46tBfzYxCk2rEcAHB%2B2BYM%3D&reserved=0</a><br>
FAQ: <a
href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=1UinGzeqP9y8I42N0m2pnTtSYk%2BCzpN7SWuhPhGyhLs%3D&reserved=0"
moz-do-not-send="true">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C3f366968cc6848e50dfc08d7299137f4%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637023576067773885&sdata=1UinGzeqP9y8I42N0m2pnTtSYk%2BCzpN7SWuhPhGyhLs%3D&reserved=0</a><br>
<br>
</div>
</span></font></div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
</body>
</html>