<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxmsonormal, li.xxmsonormal, div.xxmsonormal
{mso-style-name:x_xmsonormal;
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxxmsonormal, li.xxxmsonormal, div.xxxmsonormal
{mso-style-name:x_xxmsonormal;
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:961233504;
mso-list-template-ids:1494927964;}
@list l1
{mso-list-id:1616712464;
mso-list-type:hybrid;
mso-list-template-ids:1811596912 994995718 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-start-at:2;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:20.25pt;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:56.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:92.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:128.25pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:164.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:200.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:236.25pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:272.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:308.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:2080788096;
mso-list-template-ids:-1484612044;}
@list l2:level1
{mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#7030A0">Hi Gabor,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">New developments, on this.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">After another test this time with no remote destinations configured, the issue happened again:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">In attachment I am sending:<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="color:#7030A0;margin-left:-15.75pt;mso-list:l1 level1 lfo4">
The stats in error condition: 42116.inerror.stats.txt, where <span style="font-family:"Courier New"">
‘src.journald;s_src#0;journal;a;stamp;1651142246’</span><o:p></o:p></li><li class="MsoListParagraph" style="color:#7030A0;margin-left:-15.75pt;mso-list:l1 level1 lfo4">
The stats in error condition after some time: 42116.inerror.15m.stats.txt where <span style="font-family:"Courier New"">
‘src.journald;s_src#0;journal;a;stamp;1651142246’ </span><o:p></o:p></li><li class="MsoListParagraph" style="color:#7030A0;margin-left:-15.75pt;mso-list:l1 level1 lfo4">
The stats after recovering the system with reload: 42116.after.stats.txt<o:p></o:p></li><li class="MsoListParagraph" style="color:#7030A0;margin-left:-15.75pt;mso-list:l1 level1 lfo4">
The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf<o:p></o:p></li></ul>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">It seems the </span><span style="font-family:"Courier New";color:#7030A0">‘src.journald;s_src#0;journal;a;stamp;1651142246’</span><span style="color:#7030A0">did not changed. Does this means that last read timestamp
from journal did not changed?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Logs from the UDP source are still being written to the /var/log/… files.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Please let me know if you find something else.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Thanks & Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Alex<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> <b>
On Behalf Of </b>Alexandre Santos<br>
<b>Sent:</b> 28 de abril de 2022 11:18<br>
<b>To:</b> Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#7030A0">Hi Gabor,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Thanks for your support on this.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">I think what's even more troubling is that the other syslog-ng instance that's sending to Grafana dropped 2,2M messages:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">> The server (Grafana) is not working. It is pingable, but not receiving logs. This is one of the test conditions when the issue happens.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:black">The issue is that there are no messages from system() and internal() in the /var/log/... output files, right?</span></b><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">> Right.</span><span style="color:black"><br>
So I can assume that no messages from syslog() source go out on the d_mgmt_vrf_socket() destination when the issue happens?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">> Hum, I never check that as matter of fact, I was more focused on the /var/log/... output files.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">We will take in consideration your suggestions, to increase the disk buffer as much as possible and reduce the reconnect time.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Please let us know if you find something else.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Thanks & Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Alex<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>
<br>
<b>Sent:</b> 28 de abril de 2022 09:34<br>
<b>To:</b> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: Local sources seem not to be working<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Hi,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Sorry for not replying earlier.<br>
Thanks for the stats output, I've checked it.<br>
It looks like messages were dropped, almost 100k messages.<br>
dst.unix-stream;d_mgmt_vrf_socket#0;unix-stream,localhost.afunix:/dev/uds_log;a;dropped;95102
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">dst.unix-stream;d_mgmt_vrf_socket#0;unix-stream,localhost.afunix:/dev/uds_log;a;processed;3868284<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">dst.unix-stream;d_mgmt_vrf_socket#0;unix-stream,localhost.afunix:/dev/uds_log;a;queued;0<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">dst.unix-stream;d_mgmt_vrf_socket#0;unix-stream,localhost.afunix:/dev/uds_log;a;written;3777905<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">I think what's even more troubling is that the other syslog-ng instance that's sending to Grafana dropped 2,2M messages:<br>
dst.syslog;d_grafana_tcp#0;tcp,10.100.71.73:1514;a;dropped;2219335 <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">dst.syslog;d_grafana_tcp#0;tcp,10.100.71.73:1514;a;processed;4090485<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">dst.syslog;d_grafana_tcp#0;tcp,10.100.71.73:1514;a;queued;3753<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">dst.syslog;d_grafana_tcp#0;tcp,10.100.71.73:1514;a;suppressed;24682<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">dst.syslog;d_grafana_tcp#0;tcp,10.100.71.73:1514;a;written;1874813<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">I've tried reproducing the issue, I've placed the disk-buffer files on a tmpfs filesystem (with limited size, so I can test what happens if the disk becomes full).<br>
I was unable to reproduce the issue. For me, both the system() and internal() sources were continuing to read new messages, but since the disk-buffer is full, the messages were dropped.<br>
I've asked for some help from the team, as I have got other priorities now, so someone else will be looking into this deeper.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">I think we could narrow down the issue once we have reproduction steps, but I know this is not easy.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">The biggest mistery for me is still, how could the syslog() UDP!!! source forward messages () when system() and internal() doesn't.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">The second is how could messages go out from d_mgmg_vrf_socket if the destination is full? Maybe this isn't what is happening, so let's clear out what is really happening when "local sources doesn't work".<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">From an earlier mail:<br>
> You're saying that the syslog() source in s_src is receiving message, while the internal() and system() doesn't?
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">> [Alexandre Santos] Yes, I think that it is what is happening. Logs from syslog() source are being written to the /var/logs/..., while journald logs are not.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black"><br>
<b>The issue is that there are no messages from system() and internal() in the /var/log/... output files, right?</b><br>
So I can assume that no messages from syslog() source go out on the d_mgmt_vrf_socket() destination when the issue happens?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><br>
<b>Suggestions</b>:<o:p></o:p></span></p>
</div>
<div>
<ol start="1" type="1">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;margin-bottom:12.0pt;mso-list:l2 level1 lfo3">
An important thing to emphasize is that your config clearly shows that the disk-buffer is
<b>extremely small</b>, especially without flow-control!!!!!<br>
I strongly recommend adjusting it's size from 4MB to t least a 1GB size!!!!<br>
(It's a different question that storing a disk-buffer on a tmpfs filesystem (which is RAM based) is not really persistent, but that can be okay as long as the system runs; it's a step better than using only in-memory queue in syslog-ng)<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level1 lfo3">
You could lower the value of the reconnect timeout from the default 60 seconds to 10 seconds (just a guess), so it will be a lower window of time when messages have to be queued.<o:p></o:p></li></ol>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Regards,<br>
Gabor<br>
<br>
<br>
Gabor<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>><br>
<b>Sent:</b> Thursday, April 21, 2022 15:22<br>
<b>To:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> RE: Local sources seem not to be working</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not follow guidance,
click links, or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="xmsonormal"><span style="color:#7030A0">Hi Gabor,</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">The problem was reproduced again.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">I was able to get the stats in the error situation:
<b>54146.stats.error.txt</b></span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">I also took the stats after reloading the configuration (which fixes the problem): 54146.stats.after.txt</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">Regarding your question: </span>
Just a random question: is /tmp a tmpfs filesystem?<o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">Yes it is.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">Let me know what you found out.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">Thanks and regards,</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#7030A0">Alex</span><o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="xmsonormal"><b>From:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>
<br>
<b>Sent:</b> 31 de março de 2022 11:14<br>
<b>To:</b> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: Local sources seem not to be working<o:p></o:p></p>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<p class="xmsonormal"><span style="color:black">Thanks for the config! I'll continue experimenting on my ideas<br>
<br>
You could either configure syslog-ng-ctl stats to talk to a given syslog-ng instance with the --control option pointing to the control-socket e.g. as above /var/lib/syslog-ng/mgmt-syslog-ng.ctl,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">OR use the syslog-ng-ctl instance under the 2<sup>nd</sup> syslog-ng installation path.<br>
<br>
Just a random question: is /tmp a tmpfs filesystem?</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">Gabor</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>><br>
<b>Sent:</b> Wednesday, March 30, 2022 17:07<br>
<b>To:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> RE: Local sources seem not to be working</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="xmsonormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not follow guidance,
click links, or open attachments unless you recognize the sender and know the content is safe.</span><o:p></o:p></p>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<div>
<p class="xxmsonormal"><span style="color:#7030A0">Hi Gabor,</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0">Thank you for you feedback.</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxmsonormal"><i><span style="color:black">Can you share the config, when the issue cannot be seen?</span></i><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0">I am sending the configuration in attachment.</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxmsonormal"><i><span style="color:black">I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.</span></i><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0">The issue is hard to reproduce, the next time the error is seen, I try to run it.</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0">But how can I run syslog-ng-ctl stats for the 2<sup>nd</sup> syslog-ng instance?</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="font-size:8.0pt;font-family:Consolas;color:#7030A0">root@machine:/~# ps -ewfH | grep syslog-ng</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="font-size:8.0pt;font-family:Consolas;color:#7030A0">root 2582 1 0 09:06 ? 00:00:34 /usr/sbin/syslog-ng -F --caps cap_net_bind_service,cap_net_broadcast,cap_net_raw,cap_dac_read_search,cap_chown,cap_fowner=p
cap_dac_override,cap_syslog=ep</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="font-size:8.0pt;font-family:Consolas;color:#7030A0">root 4018 1 0 09:07 ? 00:00:00 /usr/sbin/syslog-ng -F --cfgfile=/etc/syslog-ng/mgmt-syslog-ng.conf --pidfile=/var/lib/syslog-ng/mgmt-syslog-ng.pid --persist-file=/var/lib/syslog-ng/mgmt-syslog-ng.persist
--control=/var/lib/syslog-ng/mgmt-syslog-ng.ctl</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0">syslog-ng-ctl, seems to only show stats for job 2582.</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0">Regards, Alex</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="xxmsonormal"><b>From:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>
<br>
<b>Sent:</b> 29 de março de 2022 12:48<br>
<b>To:</b> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: Local sources seem not to be working<o:p></o:p></p>
</div>
</div>
<p class="xxmsonormal"> <o:p></o:p></p>
<div>
<p class="xxmsonormal"><span style="color:black">Hi Alex,</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black">Using regular disk-buffer vs. using reliable disk-buffer shouldn't cause symptoms like that. It sounds like reliable(yes) would turn on a flow-control-like behaviour, which it doesn't.</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black">(And as you said it only affects local sources).<br>
The main difference between the two kinds of disk-buffers is, that while reliable disk-buffer write every message to the disk-buffer, a normal disk-buffer has memory-only buffers for performance reasons (and flow-control reasons too).</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black">You can still lose logs with a reliable disk-buffer if no flow-control is used: when the disk-buffer has reached it's maximum size and new messages keep arriving, then syslog-ng drops those messages.</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black">We have more detailed documentation about disk-buffers in the admin guide, where you can see the structure of disk-buffers:<br>
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.36%2Fadministration-guide%2F61%23TOPIC-1768724&data=05%7C01%7Casantos%40infinera.com%7C88bdc085cb36483408ef08da29005938%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637867378734838251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=gKf1x5RF7gadIsCanAiPxbSsV8u5MCs0WGjHW29XoEQ%3D&reserved=0">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724</a></span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black">Can you share the config, when the issue cannot be seen?</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black">I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black"><br>
Regards,</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="color:black">Gabor</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="xxmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>><br>
<b>Sent:</b> Monday, March 28, 2022 13:45<br>
<b>To:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> RE: Local sources seem not to be working</span> <o:p></o:p></p>
<div>
<p class="xxmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="xxmsonormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not follow guidance,
click links, or open attachments unless you recognize the sender and know the content is safe.</span><o:p></o:p></p>
</div>
<p class="xxmsonormal"> <o:p></o:p></p>
<div>
<div>
<p class="xxxmsonormal"><span style="color:#7030A0">Hi Gabor,</span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">“<i>This is strange: the d_localfile destinations (<span style="background:white">as well as the vrf-socket destination "d_mgmt_vrf_socket"</span>) receive messages from the syslog() source, but not from the
internal() or system() sources?”</i></span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">Yes.</span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxxmsonormal"><i><span style="color:#7030A0">And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?</span></i><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">Yes.</span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">I could not test the 2 last suggestions that you made.</span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">We did however another test, which was to
<b>remove the reliable</b> option from d_mgmt_vrf_socket, and it seems the <b>problem is not seen again</b>.</span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">Besides from what it is written in the manual, in other which cases/conditions can syslog-ng loose logs?</span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td colspan="2" style="padding:.75pt .75pt .75pt .75pt">
<p class="xxxmsonormal"><span style="color:#7030A0">reliable() </span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="xxxmsonormal"><span style="color:#7030A0">Type:</span><o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="xxxmsonormal"><span style="color:#7030A0">yes|no</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="xxxmsonormal"><span style="color:#7030A0">Default:</span><o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="xxxmsonormal"><span style="color:#7030A0">no</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td colspan="2" style="padding:.75pt .75pt .75pt .75pt">
<p class="xxxmsonormal"><span style="color:#7030A0">Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is
created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="xxxmsonormal"><span style="color:#7030A0"> </span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">Thanks in advance,</span><o:p></o:p></p>
<p class="xxxmsonormal"><span style="color:#7030A0">Alex</span><o:p></o:p></p>
<p class="xxxmsonormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="xxxmsonormal"><b>From:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>
<br>
<b>Sent:</b> 25 de março de 2022 14:44<br>
<b>To:</b> Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>>; Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: Local sources seem not to be working<o:p></o:p></p>
</div>
</div>
<p class="xxxmsonormal"> <o:p></o:p></p>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="xxxmsonormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe.</span><o:p></o:p></p>
</div>
<p class="xxxmsonormal"> <o:p></o:p></p>
<div>
<div>
<p class="xxxmsonormal"><span style="color:black">Hi Alex,</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black"><br>
Sorry I haven't answered yet. I'll have a few ideas I would like to try out next week.</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black"><br>
This is strange: the d_localfile destinations (<span style="background:white">as well as the vrf-socket destination "d_mgmt_vrf_socket"</span>) receive messages from the syslog() source, but not from the internal() or system() sources?</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black">And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black">If it would be soft flow-control, then the syslog() source would be suspended too.</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black;background:white">Just a tip: would you switch out the unix-dgram() destination to syslog() destination, please? Maybe that's not possible with the VRF in-place...</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black"><br>
In the stats output, do you see an increased number of dropped messages?</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black">I would still suggest increasing the 4MB disk-buffer. You should make an estimation of how long could the mgmt syslog-ng be down (i.e not receiving from the unix-dgram), what is the average incoming EPS and
an average message size, that could give a hint about the required disk-buffer size.</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="color:black"><br>
Regards,</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal" style="margin-bottom:12.0pt"><span style="color:black">Gabor</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>