<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I don't use Kafka, but recall, that there were some Kafka and buffering related fixes in 3.28. You should give it a try and see if it fixes your problem.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Peter<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div>
<div>
<div dir="ltr">Peter Czanik (CzP) <peter.czanik@oneidentity.com><br>
Balabit (a OneIdentity company) / syslog-ng upstream<br>
<a href="https://syslog-ng.com/community/" target="_blank">https://syslog-ng.com/<wbr>community/</a><br>
<a href="https://twitter.com/PCzanik" target="_blank">https://twitter.com/PCzanik</a></div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Ronald Fenner <rfenner@gamecircus.com><br>
<b>Sent:</b> Sunday, August 2, 2020 18:56<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] Deadlock with full disk buffer</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="">I've run into a problem where we had enabled reliable disk buffering on our Kafka destination and it's disk buffer eventually filled the drive it's stored on party because the drive is sized to the buffer it's holding.</div>
<div class=""><br class="">
</div>
<div class="">When restarting syslog-ng it complains with "Daemon exited due to a deadlock/signal/failure, restarting; exitcode='6'"</div>
<div class=""><br class="">
</div>
<div class="">The version we are using is 3.23.1</div>
<div class=""><br class="">
</div>
<div class="">I'm not sure why the disk buffer filled up the only thing I can think of is that syslog-ng was unable to send message to the Kafka destination, however our Kafka cluster is up and function fine and I can even consume messages from a topic that
we are pushing messages to on one of the instances affected.</div>
<div class=""><br class="">
</div>
<div class="">A question once a reliable message is sent is it removed from the buffer, i assume that's the case.</div>
<div class=""><br class="">
</div>
<div class="">The other question is it possible the java Kafka plugin can't keep up with the amount of mesages we are sending though syslog-ng.</div>
<div class=""><br class="">
</div>
<div class="">We see 12.5 million requests per day to the servers with the disk buffers. Each request generates at least 2 messages, one access log message, one application log so that's roughly 25 million messages a day.</div>
<div class=""><br class="">
</div>
<div class="">I designed the disk buffers to hold roughly 2 days worth of data if for some reason the Kafka cluster went down which in this case doesn't seem to be the case since monitoring tools show it's online and none of our hourly consumers complained
about being able to connect.</div>
<div class=""><br class="">
</div>
<div class="">I'll have to build a recovery server for the buffers since they are housed on attached drives the drive are preserved when the instance is terminated this was done to make sure we didn't lose any messages not sent when the autoscaling scaled out
a instance. I'm currently manually terminating the affected instances to keep those buffer and restore out logging.</div>
<div class=""><br class="">
</div>
<div class="">Once I have it, if more info is needed I can see what happens with trying to start syslog-ng up when it's also not receiving a bunch of messages as well.</div>
<div class=""><br class="">
</div>
<div class="">
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-variant-east-asian:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; word-wrap:break-word">
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-variant-east-asian:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; word-wrap:break-word">
<div class="" style="color:rgb(0,0,0); font-family:Helvetica; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-variant-east-asian:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; word-wrap:break-word">
<div class="">
<div class=""><br class="">
</div>
<div class="">Ronald Fenner</div>
<div class="">Network Architect</div>
<div class="">Game Circus LLC.</div>
</div>
<div class=""><br class="">
</div>
<div class=""><a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a></div>
</div>
</div>
</div>
</div>
<br class="">
</div>
</div>
</body>
</html>