<div dir="ltr"><div>Huge thanks to Gabor and the Balabit syslog-ng team :-)</div><div><br></div><div><a href="https://drive.google.com/drive/folders/1iJPN7lAzZtvZ9f5g-n8u7pgQ2Rwor5ze">https://drive.google.com/drive/folders/1iJPN7lAzZtvZ9f5g-n8u7pgQ2Rwor5ze</a></div><div><br></div><div>perf.data.1 and 2 and 3 were taken at a 99Hz sampling rate during 30-second windows</div><div>perf.data.4 was taken at a 99Hz sampling rate during a 5-minute window</div><div><br></div><div>*.svgs are flame graphs</div><div><br></div><div>Despite having installed glib, libc and kernel symbol packages (a no errors about missing symbols) I still see some 'unknowns'. I can dig further, but even if you see an unknown then another function at the same level or a child function should still tell you who called that.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 28, 2018 at 2:51 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div dir="auto">You should thank to Gabor and the Balabit syslog-ng team :) </div><div dir="auto"><br></div><div dir="auto">One additional ask if i may:</div><div dir="auto"><br></div>There are a number of allocations in the amqp destinations i don't like, which could be responsible for that high cpu usage. But to be sure can you pls send a fresh perf record output? Would be great to have it with debugging symbols and call graph.<div dir="auto"><br></div><div dir="auto">Thanks in advance</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mar 28, 2018 16:47, "Michal Purzynski" <<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I went ahead and installed syslog-ng from github's master with the changes from the PR. Results are nothing short of impressive.<br></div><div><br></div><div>18503 root 20 0 2859872 43580 5800 S 39.9 0.1 12:33.79 syslog-ng <wbr> <br></div><div><br></div><div>:-)</div><div><br></div><div>Syslog-ng started with fierce, taking 4.5GB and 460% (!!) of CPU then it settled down and the memory usage started shrinking quickly.</div><div><br></div><div>We are forwarding 3500 eps with 38% or so of a single CPU and 44MB of RAM.<br></div><div> <br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 28, 2018 at 11:46 AM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Awesome! Do you want me to test it? I guess I can fetch the PR and deploy the RPM in place of the affected package we have now.<br></div><div class="m_1327483693744025763m_5750692916719574179HOEnZb"><div class="m_1327483693744025763m_5750692916719574179h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 28, 2018 at 9:04 AM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">There's a potential fix here: <a href="https://github.com/balabit/syslog-ng/pull/1946" target="_blank">https://github.com/balabit/sys<wbr>log-ng/pull/1946</a><span class="m_1327483693744025763m_5750692916719574179m_1403469009482636321HOEnZb"><font color="#888888"><br><br></font></span></div><div class="gmail_extra"><span class="m_1327483693744025763m_5750692916719574179m_1403469009482636321HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_1327483693744025763m_5750692916719574179m_1403469009482636321m_-7401849877689742160gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div>
<br></font></span><div class="gmail_quote"><div><div class="m_1327483693744025763m_5750692916719574179m_1403469009482636321h5">On Wed, Mar 28, 2018 at 10:29 AM, Nagy, Gábor <span dir="ltr"><<a href="mailto:gabor.nagy@balabit.com" target="_blank">gabor.nagy@balabit.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_1327483693744025763m_5750692916719574179m_1403469009482636321h5"><div dir="ltr"><div class="gmail_extra"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I'm resending this letter to the mail list as we hit the message size limit. :)</span></div><div class="gmail_extra"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div class="gmail_extra"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Hello Michal!</span><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">I have some good news.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">We have found the reason for the increasing memory usage of syslog-ng in amqp() driver.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">It is due to our internal memory handling method (aka scratch_buffers) which we use in performance critical paths to prevent the overhead of memory allocations/deallocations of GStrings.<br>It is using memory from a pre-allocated pool instead of allocating/freeing.<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">As I have described before, the memory usage only increases if the number of queued messages reaches a new maximum.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">It is not per message and does not happen every for every queued message.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">We have several options how to fix this issue and will soon share one.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Regards,</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Gabor</div><br></div></div>
<br></div></div><span>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></span></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div></div>
</div></div></blockquote></div><br></div>