<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxmsonormal, li.xxmsonormal, div.xxmsonormal
{mso-style-name:x_xmsonormal;
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#7030A0">Hi Gabor,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Thanks for the follow up and check my answers
<b>bellow in inline with my last email</b>.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Some more details about the setup and another test it was done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">The system is running two syslog-ng instances, one in the default VRF and other in an Outer VRF.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">syslog-ng -------------- uds socket ------------------> mgmt-syslog-ng -------- UDP ---------> [Log Server]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">The syslog-ng in the default VRF is sending logs to the syslog-ng running in the outer VRF via Unix Domain Socket (destination d_mgmt_vrf_socket).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">The mgmt-syslog-ng is running in the outer VRF and sending logs to the outside world.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Only the syslog-ng in the default VRF is reading sources internal and system.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:#7030A0">We tested without having the remote logging (destination d_mgmt_vrf_socket) in the syslog-ng, and the problem did not appeared.<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:#7030A0"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:#7030A0">Hope this can give some enlightening about the problem.<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Thanks & Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0">Alex<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>
<br>
<b>Sent:</b> 17 de março de 2022 20:09<br>
<b>To:</b> Alexandre Santos <ASantos@infinera.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: Local sources seem not to be working<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">You are right, there is no flow-control for the log path where d_mgmt_vrf_socket destination is, I'm sorry.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Still, the internal log messages that the disk-buffer of d_mgmt_vrf_socket is filled are correct, but the source is not suspended.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">I have some trouble understanding the problem, can you explain it please?<br>
You're saying that the syslog() source in s_src is receiving message, while the internal() and system() doesn't?</span><o:p></o:p></p>
<p class="MsoNormal"><b><i><span style="color:#7030A0">[Alexandre Santos] Yes, I think that it is what is happening. Logs from syslog() source are being written to the /var/logs/…, while journald logs are not.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="color:black">You've also stated that journald logs are working fine.<br>
Does that mean that you can see new logs in journal, but not in syslog-ng?<o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="color:#7030A0">[Alexandre Santos] Yes.</span></i></b><span style="color:#7030A0"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">When the issue happens, can you check that internal() is working, e.g. by turning on and off the verbosity logging with "syslog-ng-ctl verbose --set on" and then "sbin/syslog-ng-ctl verbose --set off", please?<o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="color:#7030A0">[Alexandre Santos] I saw no logs when I did this in error condition so I assume internal is not working as well.</span></i></b><span style="color:#7030A0"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><br>
This would generate an internal message with info level.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Also, can you check system() source as well with the "logger" command, e.g. "logger --rfc3164 test syslog-ng", please?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black">Could you give us a syslog-ng-ctl stats output too, please?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><i><span style="color:#7030A0">[Alexandre Santos] I have to this in the next test iteration.</span></i></b><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black"><br>
<br>
Maybe I have found something, but I have to double-check: it looks like internal() source's messages are suppressed due to the destination <span style="background:white">d_mgmt_vrf_socket is unreachable:</span><br>
<44>1 2022-03-11T11:52:45.313+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="4"] internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='Suppressing
duplicate message; host=\'xmm4-1-1\', msg=\'Destination reliable queue full, dropping message; filename=\\'/tmp/syslog-ng-00016.rqf\\', queue_len=\\'6063\\', mem_buf_size=\\'2097152\\', disk_buf_size=\\'4194304\\', persist_name=\\'afsocket_dd_qfile(stream,localhost.afunix:/dev/uds_log)\\'\''<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black">This means that there are no internal() logs until the destination is not reachable again.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Gabor<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Alexandre Santos <</span><a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a><span style="color:black">><br>
<b>Sent:</b> Wednesday, March 16, 2022 16:53<br>
<b>To:</b> Gabor Nagy (gnagy) <</span><a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a><span style="color:black">>; Syslog-ng users' and developers' mailing list <</span><a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><span style="color:black">><br>
<b>Subject:</b> RE: Local sources seem not to be working</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not follow guidance,
click links, or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="xmsonormal">Hi Gabor,<o:p></o:p></p>
<p class="xmsonormal">Thanks for the feedback.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">But the flags(flow-control); is not set for the destination d_mgmt_vrf_socket. Only for the other destinations… d_localfile_<filename>.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">That also does not explain the fact that log messages from:<o:p></o:p></p>
<p class="xmsonormal">syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no));
<o:p></o:p></p>
<p class="xmsonormal">are still being written to the d_localfile_<filename>.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">Any other idea?<o:p></o:p></p>
<p class="xmsonormal">Thanks in advance,<o:p></o:p></p>
<p class="xmsonormal">Alex<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="xmsonormal"><b>From:</b> Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>>
<br>
<b>Sent:</b> 16 de março de 2022 15:09<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>>; Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>><br>
<b>Subject:</b> Re: Local sources seem not to be working<o:p></o:p></p>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<p class="xmsonormal"><span style="color:black">Hi Alex!<br>
<br>
I've checked the attached config and logs, and it looks like syslog-ng cannot send logs to the "/dev/uds_log" destination, and you have flow-control enabled in the config.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black">Once you fill the disk-buffer (which is a 4MiB sized reliable disk-buffer), flow-control kicks in and syslog-ng stops reading more messages from the sources that are connected to this destination.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"><br>
example log:<br>
Destination reliable queue full, dropping message; filename='/tmp/syslog-ng-00016.rqf', queue_len='6063', mem_buf_size='2097152', disk_buf_size='4194304', persist_name='afsocket_dd_qfile(stream,localhost.afunix:/dev/uds_log)'<br>
<br>
At first, I would suggest to increase the disk-buffer size.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal" style="margin-bottom:12.0pt"><span style="color:black">Regards,<br>
Gabor</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> syslog-ng <</span><a href="mailto:syslog-ng-bounces@lists.balabit.hu">syslog-ng-bounces@lists.balabit.hu</a><span style="color:black">> on behalf of Alexandre Santos
<</span><a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a><span style="color:black">><br>
<b>Sent:</b> Tuesday, March 15, 2022 16:04<br>
<b>To:</b> </span><a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><span style="color:black"> <</span><a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><span style="color:black">><br>
<b>Subject:</b> [syslog-ng] Local sources seem not to be working</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="xmsonormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not follow guidance,
click links, or open attachments unless you recognize the sender and know the content is safe.</span><o:p></o:p></p>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<div>
<p class="xxmsonormal">Hi,<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<p class="xxmsonormal">I have syslog-ng 3.32.1 running in a Debian GNU/Linux 10 (buster) with the configuration in the attachement.<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<p class="xxmsonormal">After sometime running, syslog-ng seems be unable to read from
<span style="font-family:Consolas">system()</span> and <span style="font-family:Consolas">
internal()</span> sources.<o:p></o:p></p>
<p class="xxmsonormal">Log messages from <span style="font-family:Consolas">syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no));</span> are seen in the output folders.<o:p></o:p></p>
<p class="xxmsonormal">Also journald logs are working fine.<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<p class="xxmsonormal">After a reload of configuration in which what changes is this line:<o:p></o:p></p>
<p class="xxmsonormal"><span style="font-family:Consolas">rewrite r_host { set("MACHINE-${HOST}", value("HOST")); };</span><o:p></o:p></p>
<p class="xxmsonormal">logging is resumed.<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<p class="xxmsonormal">Here is the time gap for logs:<o:p></o:p></p>
<p class="xxmsonormal"><span style="font-family:Consolas"><43>1 2022-03-11T11:55:23.802+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="767"] Last message 'Destination reliable' repeated 8933 times, suppressed by syslog-ng on xmm4-1-1</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="font-family:Consolas"><46>1 2022-03-14T07:19:01.817+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="1"] Module loaded and initialized successfully; module='syslogformat'</span><o:p></o:p></p>
<p class="xxmsonormal"><span style="font-family:Consolas"> </span><o:p></o:p></p>
<p class="xxmsonormal">Do you know why this is happening?<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<p class="xxmsonormal">Thanks & Regards,<o:p></o:p></p>
<p class="xxmsonormal">Alex<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>