<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div>Could you open an issue at github?</div>
<div>Do you have a test system where you can install a not released syslog-ng? (We can provide you rpm or deb packages.)</div>
<div>If you have a test system, you could replace kafka with file destination to see if the bottleneck is at kafka (or maybe at the source, or other logpipe element). Or just run the perf tool... </div>
<div><br>
</div>
<div>L.</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Ronald Fenner <rfenner@gamecircus.com><br>
<b>Sent:</b> Thursday, August 6, 2020 1:47 AM<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] Kafka performance</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="">I'm trying to figure out why the performance of syslog-ng sending to kafka has tanked considerably. Everything was running fine up until 7/30 and then all of sudden syslog-ng is having problems pushing messages to kafka at a quick pace.</div>
<div class=""><br class="">
</div>
<div class="">I've updated to 3.28.1 and it's still having problems. I was using disk buffering but I removed it an d doesn't seem to be the bottleneck.</div>
<div class=""><br class="">
</div>
<div class="">I feel it's something with syslog ng as I can use the kafka perf test to push messages at a very quick rate to the cluster. I was able to push 1 million 1k test messages in several seconds. Syslog-ng seems to be able to only push a couple of messages
a second.</div>
<div class=""><br class="">
</div>
<div class="">Any thoughts on what could be tanking it's performance? It's running on Amazon Linux 2.</div>
<div class=""><br class="">
</div>
<div class="">I've rebuilt our base image with the latest version so we can switch between kafka-c and the java version however since the kafka-c topic can't be templated at the moment until a pull request is released I can't switch to it to see if it fairs
better.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="color:rgb(0,0,0); orphans:2; widows:2">Ronald Fenner</span></div>
<div class="">
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-variant-east-asian:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; word-wrap:break-word">
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-variant-east-asian:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; word-wrap:break-word">
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-variant-east-asian:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; word-wrap:break-word">
<div class="">
<div class="">Network Architect</div>
<div class="">Game Circus LLC.</div>
</div>
<div class=""><br class="">
</div>
<div class=""><a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a></div>
</div>
</div>
</div>
</div>
<br class="">
</div>
</div>
</body>
</html>