<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Thanks for the config! I'll continue experimenting on my ideas<br>
<br>
You could either configure syslog-ng-ctl stats to talk to a given syslog-ng instance with the --control option pointing to the control-socket e.g. as above /var/lib/syslog-ng/mgmt-syslog-ng.ctl,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
OR use the syslog-ng-ctl instance under the 2<span><sup>nd</sup> syslog-ng installation path.</span><br>
<br>
Just a random question: is /tmp a tmpfs filesystem?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Gabor</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Alexandre Santos <ASantos@infinera.com><br>
<b>Sent:</b> Wednesday, March 30, 2022 17:07<br>
<b>To:</b> Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> RE: Local sources seem not to be working</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Consolas}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle23
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="color:#7030A0">Hi Gabor,</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">Thank you for you feedback.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><i><span style="color:black">Can you share the config, when the issue cannot be seen?</span></i></p>
<p class="x_MsoNormal"><span style="color:#7030A0">I am sending the configuration in attachment.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><i><span style="color:black">I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.</span></i></p>
<p class="x_MsoNormal"><span style="color:#7030A0">The issue is hard to reproduce, the next time the error is seen, I try to run it.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">But how can I run syslog-ng-ctl stats for the 2<sup>nd</sup> syslog-ng instance?</span></p>
<p class="x_MsoNormal"><span style="font-size:8.0pt; font-family:Consolas; color:#7030A0">root@machine:/~# ps -ewfH | grep syslog-ng</span></p>
<p class="x_MsoNormal"><span style="font-size:8.0pt; font-family:Consolas; color:#7030A0">root 2582 1 0 09:06 ? 00:00:34 /usr/sbin/syslog-ng -F --caps cap_net_bind_service,cap_net_broadcast,cap_net_raw,cap_dac_read_search,cap_chown,cap_fowner=p
cap_dac_override,cap_syslog=ep</span></p>
<p class="x_MsoNormal"><span style="font-size:8.0pt; font-family:Consolas; color:#7030A0">root 4018 1 0 09:07 ? 00:00:00 /usr/sbin/syslog-ng -F --cfgfile=/etc/syslog-ng/mgmt-syslog-ng.conf --pidfile=/var/lib/syslog-ng/mgmt-syslog-ng.pid --persist-file=/var/lib/syslog-ng/mgmt-syslog-ng.persist
--control=/var/lib/syslog-ng/mgmt-syslog-ng.ctl</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">syslog-ng-ctl, seems to only show stats for job 2582.</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"><span style="color:#7030A0">Regards, Alex</span></p>
<p class="x_MsoNormal"><span style="color:#7030A0"> </span></p>
<p class="x_MsoNormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal"><b>From:</b> Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>
<br>
<b>Sent:</b> 29 de março de 2022 12:48<br>
<b>To:</b> Alexandre Santos <ASantos@infinera.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: Local sources seem not to be working</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal"><span style="color:black">Hi Alex,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">Using regular disk-buffer vs. using reliable disk-buffer shouldn't cause symptoms like that. It sounds like reliable(yes) would turn on a flow-control-like behaviour, which it doesn't.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">(And as you said it only affects local sources).<br>
The main difference between the two kinds of disk-buffers is, that while reliable disk-buffer write every message to the disk-buffer, a normal disk-buffer has memory-only buffers for performance reasons (and flow-control reasons too).</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">You can still lose logs with a reliable disk-buffer if no flow-control is used: when the disk-buffer has reached it's maximum size and new messages keep arriving, then syslog-ng drops those messages.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">We have more detailed documentation about disk-buffers in the admin guide, where you can see the structure of disk-buffers:<br>
<a href="https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.36%2Fadministration-guide%2F61%23TOPIC-1768724&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7C0d85939ee3414be620ac08da125eff40%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637842496489115356%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RuZsQ3d5rVGNBzd1uP8yzHHqKYM5E3Z%2FkThc8F%2Byd7g%3D&reserved=0" originalsrc="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724" shash="As/73FXrCidWjmGmYoJHlRlqv0rEu3TEZMHC95S12jUUO90HcFXwEuzFi2z+CZ8hyncWbwKJTEG3FaReSr4tNH8BfmKgKVrmFYDJK3Y6K9nwC2whLR7bl3LORoS0YftfiXef6+gDKGRSMr8Z6ZxIDoygxjyNLjNgtEMy8tvSamo=">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724</a></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">Can you share the config, when the issue cannot be seen?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black"><br>
Regards,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="color:black">Gabor</span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>><br>
<b>Sent:</b> Monday, March 28, 2022 13:45<br>
<b>To:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> RE: Local sources seem not to be working</span> </p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div style="border:solid #9C6500 1.0pt; padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="x_MsoNormal" style="line-height:12.0pt; background:#FFEB9C"><b><span style="font-size:10.0pt; color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt; color:black"> This email originated from outside of the organization. Do not follow guidance,
click links, or open attachments unless you recognize the sender and know the content is safe.</span></p>
</div>
<p class="x_MsoNormal"> </p>
<div>
<div>
<p class="x_xmsonormal"><span style="color:#7030A0">Hi Gabor,</span></p>
<p class="x_xmsonormal"><span style="color:#7030A0"> </span></p>
<p class="x_xmsonormal"><span style="color:#7030A0">“<i>This is strange: the d_localfile destinations (<span style="background:white">as well as the vrf-socket destination "d_mgmt_vrf_socket"</span>) receive messages from the syslog() source, but not from the
internal() or system() sources?”</i></span></p>
<p class="x_xmsonormal"><span style="color:#7030A0">Yes.</span></p>
<p class="x_xmsonormal"><span style="color:#7030A0"> </span></p>
<p class="x_xmsonormal"><i><span style="color:#7030A0">And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?</span></i></p>
<p class="x_xmsonormal"><span style="color:#7030A0">Yes.</span></p>
<p class="x_xmsonormal"><span style="color:#7030A0"> </span></p>
<p class="x_xmsonormal"><span style="color:#7030A0">I could not test the 2 last suggestions that you made.</span></p>
<p class="x_xmsonormal"><span style="color:#7030A0"> </span></p>
<p class="x_xmsonormal"><span style="color:#7030A0">We did however another test, which was to
<b>remove the reliable</b> option from d_mgmt_vrf_socket, and it seems the <b>problem is not seen again</b>.</span></p>
<p class="x_xmsonormal"><span style="color:#7030A0"> </span></p>
<p class="x_xmsonormal"><span style="color:#7030A0">Besides from what it is written in the manual, in other which cases/conditions can syslog-ng loose logs?</span></p>
<p class="x_xmsonormal"><span style="color:#7030A0"> </span></p>
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td colspan="2" style="padding:.75pt .75pt .75pt .75pt">
<p class="x_xmsonormal"><span style="color:#7030A0">reliable() </span></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_xmsonormal"><span style="color:#7030A0">Type:</span></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_xmsonormal"><span style="color:#7030A0">yes|no</span></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_xmsonormal"><span style="color:#7030A0">Default:</span></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_xmsonormal"><span style="color:#7030A0">no</span></p>
</td>
</tr>
<tr>
<td colspan="2" style="padding:.75pt .75pt .75pt .75pt">
<p class="x_xmsonormal"><span style="color:#7030A0">Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is
created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.</span></p>
</td>
</tr>
</tbody>
</table>
<p class="x_xmsonormal"><span style="color:#7030A0"> </span></p>
<p class="x_xmsonormal"><span style="color:#7030A0">Thanks in advance,</span></p>
<p class="x_xmsonormal"><span style="color:#7030A0">Alex</span></p>
<p class="x_xmsonormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_xmsonormal"><b>From:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>
<br>
<b>Sent:</b> 25 de março de 2022 14:44<br>
<b>To:</b> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: Local sources seem not to be working</p>
</div>
</div>
<p class="x_xmsonormal"> </p>
<div style="border:solid #9C6500 1.0pt; padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="x_xmsonormal" style="line-height:12.0pt; background:#FFEB9C"><b><span style="font-size:10.0pt; color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt; color:black"> This email originated from outside of the organization. Do not click links
or open attachments unless you recognize the sender and know the content is safe.</span></p>
</div>
<p class="x_xmsonormal"> </p>
<div>
<div>
<p class="x_xmsonormal"><span style="color:black">Hi Alex,</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black"><br>
Sorry I haven't answered yet. I'll have a few ideas I would like to try out next week.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black"><br>
This is strange: the d_localfile destinations (<span style="background:white">as well as the vrf-socket destination "d_mgmt_vrf_socket"</span>) receive messages from the syslog() source, but not from the internal() or system() sources?</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black">And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black">If it would be soft flow-control, then the syslog() source would be suspended too.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black; background:white">Just a tip: would you switch out the unix-dgram() destination to syslog() destination, please? Maybe that's not possible with the VRF in-place...</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black"><br>
In the stats output, do you see an increased number of dropped messages?</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black">I would still suggest increasing the 4MB disk-buffer. You should make an estimation of how long could the mgmt syslog-ng be down (i.e not receiving from the unix-dgram), what is the average incoming EPS and
an average message size, that could give a hint about the required disk-buffer size.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="color:black"><br>
Regards,</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin-bottom:12.0pt"><span style="color:black">Gabor</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>