<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">After more investigation including looking at the packets going between the instance and the kafka cluster I found the problem.<div class=""><br class=""></div><div class="">It appears that one of the tpoics didn't exist that messages were trying to push to. This seems like was blocking all message going to any of the other topics that existed. Once I added the topic on the cluster's side messages started being written.</div><div class=""><br class=""></div><div class="">I'm guessing syslog-ng was getting stuck trying to write to the non-existent topic and then not writing any of hte messages to topics that actually existed.<br class=""><div class=""><div class=""><br class=""></div><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">Ronald Fenner</div><div class="">Network Architect</div><div class="">Game Circus LLC.</div></div><div class=""><br class=""></div><div class=""><a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a></div></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Aug 6, 2020, at 12:01 AM, Ronald Fenner <<a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I build from source so if provided with a tar ball I can drop it into the build process and generate test image to see what happens. I know it's not an issue with it receiving the messages they are sent to a unix pipe and if I enable trace I see the messages go screaming by when I tail the log.<div class=""><br class=""></div><div class="">The problems seems to be on the sending side and I'm not sure why.</div><div class=""><br class=""></div><div class="">You mentioned a perf tool what is it so I can look for it and then see how to run it.</div><div class=""><br class=""></div><div class="">I'll also go open an issue on GitHub.</div><div class=""><br class=""><div class="">
<div style="font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class="">Ronald Fenner</div><div class="">Network Architect</div><div class="">Game Circus LLC.</div></div><div class=""><br class=""></div><div class=""><a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a></div></div></div></div>
</div>
<div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 5, 2020, at 11:51 PM, Laszlo Budai (lbudai) <<a href="mailto:Laszlo.Budai@oneidentity.com" class="">Laszlo.Budai@oneidentity.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class="">
<div class="">
<div class="">Could you open an issue at github?</div>
<div class="">Do you have a test system where you can install a not released syslog-ng? (We can provide you rpm or deb packages.)</div>
<div class="">If you have a test system, you could replace kafka with file destination to see if the bottleneck is at kafka (or maybe at the source, or other logpipe element). Or just run the perf tool... </div>
<div class=""><br class="">
</div>
<div class="">L.</div>
<hr style="display:inline-block;width:98%" tabindex="-1" class="">
<div id="divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size:11pt" class=""><b class="">From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" class="">syslog-ng-bounces@lists.balabit.hu</a>> on behalf of Ronald Fenner <<a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a>><br class="">
<b class="">Sent:</b> Thursday, August 6, 2020 1:47 AM<br class="">
<b class="">To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" class="">syslog-ng@lists.balabit.hu</a>><br class="">
<b class="">Subject:</b> [syslog-ng] Kafka performance</font>
<div class=""> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="background-color: rgb(255, 235, 156); width: 100%; border: 1pt solid rgb(156, 101, 0); padding: 2pt; font-size: 10pt; line-height: 12pt; font-family: Calibri; text-align: left;" class="">
<span style="color:#9C6500; font-weight:bold" class="">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br class="">
<div class="">
<div class="">I'm trying to figure out why the performance of syslog-ng sending to kafka has tanked considerably. Everything was running fine up until 7/30 and then all of sudden syslog-ng is having problems pushing messages to kafka at a quick pace.</div>
<div class=""><br class="">
</div>
<div class="">I've updated to 3.28.1 and it's still having problems. I was using disk buffering but I removed it an d doesn't seem to be the bottleneck.</div>
<div class=""><br class="">
</div>
<div class="">I feel it's something with syslog ng as I can use the kafka perf test to push messages at a very quick rate to the cluster. I was able to push 1 million 1k test messages in several seconds. Syslog-ng seems to be able to only push a couple of messages
a second.</div>
<div class=""><br class="">
</div>
<div class="">Any thoughts on what could be tanking it's performance? It's running on Amazon Linux 2.</div>
<div class=""><br class="">
</div>
<div class="">I've rebuilt our base image with the latest version so we can switch between kafka-c and the java version however since the kafka-c topic can't be templated at the moment until a pull request is released I can't switch to it to see if it fairs
better.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="orphans: 2; widows: 2;">Ronald Fenner</span></div>
<div class="">
<div class="" style="font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; word-wrap: break-word;">
<div class="" style="font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; word-wrap: break-word;">
<div class="" style="font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; word-wrap: break-word;">
<div class="">
<div class="">Network Architect</div>
<div class="">Game Circus LLC.</div>
</div>
<div class=""><br class="">
</div>
<div class=""><a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a></div>
</div>
</div>
</div>
</div>
<br class="">
</div>
</div>
</div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></body></html>