<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi Alex,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Using regular disk-buffer vs. using reliable disk-buffer shouldn't cause symptoms like that. It sounds like reliable(yes) would turn on a flow-control-like behaviour, which it doesn't.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
(And as you said it only affects local sources).<br>
The main difference between the two kinds of disk-buffers is, that while reliable disk-buffer write every message to the disk-buffer, a normal disk-buffer has memory-only buffers for performance reasons (and flow-control reasons too).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
You can still lose logs with a reliable disk-buffer if no flow-control is used: when the disk-buffer has reached it's maximum size and new messages keep arriving, then syslog-ng drops those messages.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
We have more detailed documentation about disk-buffers in the admin guide, where you can see the structure of disk-buffers:<br>
<a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724</a></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Can you share the config, when the issue cannot be seen?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt;">I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
Regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Gabor</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Alexandre Santos <ASantos@infinera.com><br>
<b>Sent:</b> Monday, March 28, 2022 13:45<br>
<b>To:</b> Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> RE: Local sources seem not to be working</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle19
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="color:#7030A0">Hi Gabor,</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">“<i>This is strange: the d_localfile destinations (<span style="background:white">as well as the vrf-socket destination "d_mgmt_vrf_socket"</span>) receive messages from the syslog() source, but not from the
internal() or system() sources?”</i></span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">Yes.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><i><span style="color:#7030A0">And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?</span></i></p>
<p class="x_MsoNormal"><span style="color:#7030A0">Yes.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">I could not test the 2 last suggestions that you made.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">We did however another test, which was to
<b>remove the reliable</b> option from d_mgmt_vrf_socket, and it seems the <b>problem is not seen again</b>.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">Besides from what it is written in the manual, in other which cases/conditions can syslog-ng loose logs?</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td colspan="2" style="padding:.75pt .75pt .75pt .75pt">
<p class="x_MsoNormal"><span style="color:#7030A0">reliable() </span></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_MsoNormal"><span style="color:#7030A0">Type:</span></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_MsoNormal"><span style="color:#7030A0">yes|no</span></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_MsoNormal"><span style="color:#7030A0">Default:</span></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_MsoNormal"><span style="color:#7030A0">no</span></p>
</td>
</tr>
<tr>
<td colspan="2" style="padding:.75pt .75pt .75pt .75pt">
<p class="x_MsoNormal" style=""><span style="color:#7030A0">Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option.
It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.</span></p>
</td>
</tr>
</tbody>
</table>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">Thanks in advance,</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">Alex</span></p>
<p class="x_MsoNormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal"><b>From:</b> Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>
<br>
<b>Sent:</b> 25 de março de 2022 14:44<br>
<b>To:</b> Alexandre Santos <ASantos@infinera.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: Local sources seem not to be working</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div style="border:solid #9C6500 1.0pt; padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="x_MsoNormal" style="line-height:12.0pt; background:#FFEB9C"><b><span style="font-size:10.0pt; color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt; color:black"> This email originated from outside of the organization. Do not click links
or open attachments unless you recognize the sender and know the content is safe.</span></p>
</div>
<p class="x_MsoNormal"> </p>
<div>
<div>
<p class="x_MsoNormal"><span style="color:black">Hi Alex,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"><br>
Sorry I haven't answered yet. I'll have a few ideas I would like to try out next week.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"><br>
This is strange: the d_localfile destinations (<span style="background:white">as well as the vrf-socket destination "d_mgmt_vrf_socket"</span>) receive messages from the syslog() source, but not from the internal() or system() sources?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">If it would be soft flow-control, then the syslog() source would be suspended too.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black; background:white">Just a tip: would you switch out the unix-dgram() destination to syslog() destination, please? Maybe that's not possible with the VRF in-place...</span><span style="color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"><br>
In the stats output, do you see an increased number of dropped messages?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">I would still suggest increasing the 4MB disk-buffer. You should make an estimation of how long could the mgmt syslog-ng be down (i.e not receiving from the unix-dgram), what is the average incoming EPS and an
average message size, that could give a hint about the required disk-buffer size.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"><br>
Regards,</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><span style="color:black">Gabor<br>
<br>
</span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>