<div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">I haven't seen anything like this. We are reading the journal files using libsystemd.</div><div dir="auto"><br></div><div dir="auto">Try to remove the syslog-ng persist file and check if reading the journal restarts.</div><div dir="auto"><br></div><div dir="auto">Also there's can be two ways of local messages getting to syslog-ng,</div><div dir="auto"><br></div><div dir="auto">1) /dev/log forwarding</div><div dir="auto">2) reading the journal files</div><div dir="auto"><br></div><div dir="auto">The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.</div><div dir="auto"><br></div><div dir="auto">To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.</div><div dir="auto"><br></div><div dir="auto">I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 24, 2022, 18:21 Alexandre Santos <<a href="mailto:ASantos@infinera.com">ASantos@infinera.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="m_-8312131628699672886WordSection1">
<p class="MsoNormal">Hi<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Any news regarding this issue?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Making a recap of the findings:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<ul style="margin-top:0cm" type="disc">
<li class="m_-8312131628699672886MsoListParagraph" style="margin-left:0cm">Using a Debian 10 buster with first release with 3.36.1;<u></u><u></u></li></ul>
<ul style="margin-top:0cm" type="disc">
<li class="m_-8312131628699672886MsoListParagraph" style="margin-left:0cm">After some time “system()” source logs are not getting written to the destinations;<u></u><u></u></li><li class="m_-8312131628699672886MsoListParagraph" style="margin-left:0cm">The log messages from other sources, internal() and syslog(…) continue to work fine, being written to the destinations;<u></u><u></u></li><li class="m_-8312131628699672886MsoListParagraph" style="margin-left:0cm">One the things I noticed is that the socket to the journal seems to vanish during the error situation:<u></u><u></u></li></ul>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">It seems that somehow syslog-ng in unable to read from linux journal.<u></u><u></u></p>
<p class="MsoNormal"><b>Have you ever experienced this problem?<u></u><u></u></b></p>
<p class="MsoNormal"><b>Do know what can be wrong with the system?<u></u><u></u></b></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks in advance,<u></u><u></u></p>
<p class="MsoNormal">Alex<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng-bounces@lists.balabit.hu</a>> <b>
On Behalf Of </b>Alexandre Santos<br>
<b>Sent:</b> 19 de maio de 2022 09:25<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Hi Szilard,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">There is no filter:<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">source syslog_ng_src {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> internal();<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">};<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">destination d_localfile_syslog_ng {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> flags(syslog-protocol)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> suppress(5)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> disk-buffer(<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> mem-buf-size(2097152)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> disk-buf-size(4194304)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> reliable(yes)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> dir("/tmp")<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> )<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> );<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">};<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">log {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> source(syslog_ng_src);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> destination(d_localfile_syslog_ng);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> flags(flow-control);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">};<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
<p class="MsoNormal">Alex<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng-bounces@lists.balabit.hu</a>> <b>
On Behalf Of </b>Szilard Parrag (sparrag)<br>
<b>Sent:</b> 19 de maio de 2022 08:59<br>
<b>To:</b> <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng@lists.balabit.hu</a><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:solid #9c6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#ffeb9c"><b><span style="font-size:10.0pt;color:#9c6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not click links or open
attachments unless you recognize the sender and know the content is safe.<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi Alex,<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control
<span style="background:white">enabled</span>.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Szilard<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>