<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Bryan,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for using the syslog-ng mailing list! <span id="🙂" title=":slight_smile:">
🙂</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Message drop could happen for several reasons. I couple reasons, that suddenly comes up to me:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<ol>
<li>The log path has a filter or parser, which does not match for the message.</li><li>The log path has a source, which has a built-in parser (syslog for example), and the message does not match the protocol.</li><li>Flow-control is not enabled and the destination is not alive for a longer period of time.</li><li>Flow-control is configured incorrectly.</li><li>...</li></ol>
<div>Saying so, it is hard to come up with one general way to investigate this.</div>
<div>However, I can give you some tips:</div>
<div>
<ol>
<li>You can set "stats-level(1)" in the global options and use "sbin/syslog-ng-ctl stats", then look for the "dropped" counters.</li><li>You can start syslog-ng in debug mode (./sbin/syslog-ng -Fedtv) and look for the following logs: "Destination queue full, dropping message;" or "UNMATCHED".</li><li>Check if flow-control is enabled and configured properly.</li><li>If you do not want to use flow-control, you can use disk-queue alternatively.</li></ol>
<div>If you could share some parts of your setup and config, where the problem happens, we could give you more insight.</div>
</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Attila</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Klimek, Bryan J. <bklimek@mayo.edu><br>
<b>Sent:</b> Monday, June 24, 2019 3:05 PM<br>
<b>To:</b> 'syslog-ng@lists.balabit.hu'<br>
<b>Subject:</b> [syslog-ng] Missing messages</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif"}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.x_EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="blue" vlink="purple">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal">First time poster, so be gentle.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">We run syslog-ng in our environment as a centralized syslog manager for the archiving of syslog data. We have over 2000+ Unix/Linux systems sending their syslog data with a daily ingest rate of about 10GB per day.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">It was recently pointed out to me that one particular message from one specific host is not getting persisted to the files on our syslog-ng server all the time. That is to say, it is intermittent.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">If one person can find one message that is not making into our syslog-ng archive, I can only assume that we are dropping other messages as well.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">How can I debug if and when messages are being lost and not making into the files on my centralized syslog server?</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Bryan Klimek</p>
<p class="x_MsoNormal">Mayo Clinic</p>
<p class="x_MsoNormal"> </p>
</div>
</div>
</div>
</body>
</html>