Re: [syslog-ng] Local sources seem not to be working
Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard
Hi Szilard, There is no filter: source syslog_ng_src { internal(); }; destination d_localfile_syslog_ng { program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10" flags(syslog-protocol) suppress(5) disk-buffer( mem-buf-size(2097152) disk-buf-size(4194304) reliable(yes) dir("/tmp") ) ); }; log { source(syslog_ng_src); destination(d_localfile_syslog_ng); flags(flow-control); }; Thanks and Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Szilard Parrag (sparrag) Sent: 19 de maio de 2022 08:59 To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] Local sources seem not to be working CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard
Hi Any news regarding this issue? Making a recap of the findings: * Using a Debian 10 buster with first release with 3.36.1; * After some time "system()" source logs are not getting written to the destinations; * The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations; * One the things I noticed is that the socket to the journal seems to vanish during the error situation: It seems that somehow syslog-ng in unable to read from linux journal. Have you ever experienced this problem? Do know what can be wrong with the system? root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal Thanks in advance, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Alexandre Santos Sent: 19 de maio de 2022 09:25 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working Hi Szilard, There is no filter: source syslog_ng_src { internal(); }; destination d_localfile_syslog_ng { program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10" flags(syslog-protocol) suppress(5) disk-buffer( mem-buf-size(2097152) disk-buf-size(4194304) reliable(yes) dir("/tmp") ) ); }; log { source(syslog_ng_src); destination(d_localfile_syslog_ng); flags(flow-control); }; Thanks and Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Szilard Parrag (sparrag) Sent: 19 de maio de 2022 08:59 To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] Local sources seem not to be working CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard
Hi, I haven't seen anything like this. We are reading the journal files using libsystemd. Try to remove the syslog-ng persist file and check if reading the journal restarts. Also there's can be two ways of local messages getting to syslog-ng, 1) /dev/log forwarding 2) reading the journal files The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source. To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded. I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue. On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com> wrote:
Hi
Any news regarding this issue?
Making a recap of the findings:
- Using a Debian 10 buster with first release with 3.36.1;
- After some time “system()” source logs are not getting written to the destinations; - The log messages from other sources, internal() and syslog(…) continue to work fine, being written to the destinations; - One the things I noticed is that the socket to the journal seems to vanish during the error situation:
It seems that somehow syslog-ng in unable to read from linux journal.
*Have you ever experienced this problem?*
*Do know what can be wrong with the system?*
root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
Thanks in advance,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> * On Behalf Of *Alexandre Santos *Sent:* 19 de maio de 2022 09:25 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Szilard,
There is no filter:
source syslog_ng_src {
internal();
};
destination d_localfile_syslog_ng {
program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
flags(syslog-protocol)
suppress(5)
disk-buffer(
mem-buf-size(2097152)
disk-buf-size(4194304)
reliable(yes)
dir("/tmp")
)
);
};
log {
source(syslog_ng_src);
destination(d_localfile_syslog_ng);
flags(flow-control);
};
Thanks and Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> * On Behalf Of *Szilard Parrag (sparrag) *Sent:* 19 de maio de 2022 08:59 *To:* syslog-ng@lists.balabit.hu *Subject:* Re: [syslog-ng] Local sources seem not to be working
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Alex,
We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.
Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?
Szilard
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Balazs, Thanks for the feedback! It is definitely using journald, as you can see below. When you say "Try to remove the syslog-ng persist file and check if reading the journal restarts.", this is to do when the system in the error condition, right? Regards, Alex source s_src { #Start Block source generator system channel { source { systemd-journal(); }; # source channel { channel { parser { #Start Block parser generator app-parser channel { junction { channel { filter { tags('.app.doesnotexist'); }; flags(final); }; }; } #End Block parser generator app-parser ; }; flags(final); }; channel { flags(final); }; }; }; # channel #End Block source generator system ; internal(); syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no)); }; From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Balazs Scheidler Sent: 26 de junho de 2022 06:37 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working Hi, I haven't seen anything like this. We are reading the journal files using libsystemd. Try to remove the syslog-ng persist file and check if reading the journal restarts. Also there's can be two ways of local messages getting to syslog-ng, 1) /dev/log forwarding 2) reading the journal files The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source. To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded. I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue. On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi Any news regarding this issue? Making a recap of the findings: * Using a Debian 10 buster with first release with 3.36.1; * After some time "system()" source logs are not getting written to the destinations; * The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations; * One the things I noticed is that the socket to the journal seems to vanish during the error situation: It seems that somehow syslog-ng in unable to read from linux journal. Have you ever experienced this problem? Do know what can be wrong with the system? root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> Thanks in advance, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Alexandre Santos Sent: 19 de maio de 2022 09:25 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi Szilard, There is no filter: source syslog_ng_src { internal(); }; destination d_localfile_syslog_ng { program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10" flags(syslog-protocol) suppress(5) disk-buffer( mem-buf-size(2097152) disk-buf-size(4194304) reliable(yes) dir("/tmp") ) ); }; log { source(syslog_ng_src); destination(d_localfile_syslog_ng); flags(flow-control); }; Thanks and Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Szilard Parrag (sparrag) Sent: 19 de maio de 2022 08:59 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=WNrM7LVg0OQZ4H%2FdhbZd4mCQ%2F6dAUsu9gJlgEa2aoCk%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=O5s9C1DJkKe%2F%2FJRWzDpRO4tpU1I31b%2F%2BG1SBXmG26Ps%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=CiurkG9FrrVfxETCA8L5nVWoHofbzNrwn%2BP%2BpfAHX1k%3D&reserved=0>
This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us. If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop. Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume. I don't really have any other ideas at the moment. Still on vacation. :) On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos@infinera.com> wrote:
Hi Balazs,
Thanks for the feedback!
It is definitely using journald, as you can see below.
When you say “Try to remove the syslog-ng persist file and check if reading the journal restarts.”, this is to do when the system in the error condition, right?
Regards,
Alex
*source s_src {*
*#Start Block source generator system*
*channel {*
* source {*
*systemd-journal();*
* }; # source*
*channel {*
* channel {*
* parser {*
*#Start Block parser generator app-parser*
*channel {*
* junction {*
*channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };*
*}*
*#End Block parser generator app-parser*
*;*
* };*
* flags(final);*
* };*
* channel { flags(final); };*
*};*
*}; # channel*
*#End Block source generator system*
*;*
* internal();*
* syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));*
*};*
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> * On Behalf Of *Balazs Scheidler *Sent:* 26 de junho de 2022 06:37 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi,
I haven't seen anything like this. We are reading the journal files using libsystemd.
Try to remove the syslog-ng persist file and check if reading the journal restarts.
Also there's can be two ways of local messages getting to syslog-ng,
1) /dev/log forwarding
2) reading the journal files
The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.
To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.
I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.
On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com> wrote:
Hi
Any news regarding this issue?
Making a recap of the findings:
- Using a Debian 10 buster with first release with 3.36.1;
- After some time “system()” source logs are not getting written to the destinations; - The log messages from other sources, internal() and syslog(…) continue to work fine, being written to the destinations; - One the things I noticed is that the socket to the journal seems to vanish during the error situation:
It seems that somehow syslog-ng in unable to read from linux journal.
*Have you ever experienced this problem?*
*Do know what can be wrong with the system?*
root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
Thanks in advance,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Alexandre Santos *Sent:* 19 de maio de 2022 09:25 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Szilard,
There is no filter:
source syslog_ng_src {
internal();
};
destination d_localfile_syslog_ng {
program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
flags(syslog-protocol)
suppress(5)
disk-buffer(
mem-buf-size(2097152)
disk-buf-size(4194304)
reliable(yes)
dir("/tmp")
)
);
};
log {
source(syslog_ng_src);
destination(d_localfile_syslog_ng);
flags(flow-control);
};
Thanks and Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Szilard Parrag (sparrag) *Sent:* 19 de maio de 2022 08:59 *To:* syslog-ng@lists.balabit.hu *Subject:* Re: [syslog-ng] Local sources seem not to be working
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Alex,
We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.
Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?
Szilard
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=WNrM7LVg0OQZ4H%2FdhbZd4mCQ%2F6dAUsu9gJlgEa2aoCk%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=O5s9C1DJkKe%2F%2FJRWzDpRO4tpU1I31b%2F%2BG1SBXmG26Ps%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=CiurkG9FrrVfxETCA8L5nVWoHofbzNrwn%2BP%2BpfAHX1k%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Balazs, Do you have any other idea of how to debug this?
From the last time I was able to reproduce the problem, I found the following:
New developments, on this. After another test this time with no remote destinations configured, the issue happened again: In attachment I am sending: - The stats in error condition: 42116.inerror.stats.txt, where 'src.journald;s_src#0;journal;a;stamp;1651142246' - The stats in error condition after some time: 42116.inerror.15m.stats.txt where 'src.journald;s_src#0;journal;a;stamp;1651142246' - The stats after recovering the system with reload: 42116.after.stats.txt - The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf It seems the 'src.journald;s_src#0;journal;a;stamp;1651142246'did not changed. Does this means that last read timestamp from journal did not changed? Logs from the UDP source are still being written to the /var/log/... files. Cheers, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Balazs Scheidler Sent: 1 de julho de 2022 21:02 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us. If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop. Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume. I don't really have any other ideas at the moment. Still on vacation. :) On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi Balazs, Thanks for the feedback! It is definitely using journald, as you can see below. When you say "Try to remove the syslog-ng persist file and check if reading the journal restarts.", this is to do when the system in the error condition, right? Regards, Alex source s_src { #Start Block source generator system channel { source { systemd-journal(); }; # source channel { channel { parser { #Start Block parser generator app-parser channel { junction { channel { filter { tags('.app.doesnotexist'); }; flags(final); }; }; } #End Block parser generator app-parser ; }; flags(final); }; channel { flags(final); }; }; }; # channel #End Block source generator system ; internal(); syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no)); }; From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Balazs Scheidler Sent: 26 de junho de 2022 06:37 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi, I haven't seen anything like this. We are reading the journal files using libsystemd. Try to remove the syslog-ng persist file and check if reading the journal restarts. Also there's can be two ways of local messages getting to syslog-ng, 1) /dev/log forwarding 2) reading the journal files The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source. To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded. I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue. On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi Any news regarding this issue? Making a recap of the findings: * Using a Debian 10 buster with first release with 3.36.1; * After some time "system()" source logs are not getting written to the destinations; * The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations; * One the things I noticed is that the socket to the journal seems to vanish during the error situation: It seems that somehow syslog-ng in unable to read from linux journal. Have you ever experienced this problem? Do know what can be wrong with the system? root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> Thanks in advance, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Alexandre Santos Sent: 19 de maio de 2022 09:25 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi Szilard, There is no filter: source syslog_ng_src { internal(); }; destination d_localfile_syslog_ng { program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10" flags(syslog-protocol) suppress(5) disk-buffer( mem-buf-size(2097152) disk-buf-size(4194304) reliable(yes) dir("/tmp") ) ); }; log { source(syslog_ng_src); destination(d_localfile_syslog_ng); flags(flow-control); }; Thanks and Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Szilard Parrag (sparrag) Sent: 19 de maio de 2022 08:59 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C2ffd65290fa94948ab3008da5b9c9efc%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637923025515816554%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=p6wf%2Bdk0xxwK%2FjNjHLPDnK75ASR9cNnKQrMKg2%2F4WP4%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C2ffd65290fa94948ab3008da5b9c9efc%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637923025515816554%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f4rDYxJk%2BPnY1HoaPciQBXk5MwC2EaCwe7yes1CxqiM%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C2ffd65290fa94948ab3008da5b9c9efc%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637923025515816554%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UPOtBeu3S2lcfnUxBEwxHoqmhBDZX%2FvJ5ruolgB2%2ByI%3D&reserved=0> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C2ffd65290fa94948ab3008da5b9c9efc%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637923025515816554%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=p6wf%2Bdk0xxwK%2FjNjHLPDnK75ASR9cNnKQrMKg2%2F4WP4%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C2ffd65290fa94948ab3008da5b9c9efc%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637923025515972761%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9vBmZeKe65PzA6U%2FjRqmodu1VfYLeObj86sZhnscASU%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C2ffd65290fa94948ab3008da5b9c9efc%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637923025515972761%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9Mimn31FEJM1rIMS1F0sQBGZXxrGov5rjug2JsmXaKI%3D&reserved=0>
Hi, Did you have a chance to analyze this? Thanks & Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Alexandre Santos Sent: 16 de setembro de 2022 14:38 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working Hi Balazs, Do you have any other idea of how to debug this?
From the last time I was able to reproduce the problem, I found the following:
New developments, on this. After another test this time with no remote destinations configured, the issue happened again: In attachment I am sending: - The stats in error condition: 42116.inerror.stats.txt, where 'src.journald;s_src#0;journal;a;stamp;1651142246' - The stats in error condition after some time: 42116.inerror.15m.stats.txt where 'src.journald;s_src#0;journal;a;stamp;1651142246' - The stats after recovering the system with reload: 42116.after.stats.txt - The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf It seems the 'src.journald;s_src#0;journal;a;stamp;1651142246'did not changed. Does this means that last read timestamp from journal did not changed? Logs from the UDP source are still being written to the /var/log/... files. Cheers, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Balazs Scheidler Sent: 1 de julho de 2022 21:02 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us. If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop. Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume. I don't really have any other ideas at the moment. Still on vacation. :) On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi Balazs, Thanks for the feedback! It is definitely using journald, as you can see below. When you say "Try to remove the syslog-ng persist file and check if reading the journal restarts.", this is to do when the system in the error condition, right? Regards, Alex source s_src { #Start Block source generator system channel { source { systemd-journal(); }; # source channel { channel { parser { #Start Block parser generator app-parser channel { junction { channel { filter { tags('.app.doesnotexist'); }; flags(final); }; }; } #End Block parser generator app-parser ; }; flags(final); }; channel { flags(final); }; }; }; # channel #End Block source generator system ; internal(); syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no)); }; From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Balazs Scheidler Sent: 26 de junho de 2022 06:37 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi, I haven't seen anything like this. We are reading the journal files using libsystemd. Try to remove the syslog-ng persist file and check if reading the journal restarts. Also there's can be two ways of local messages getting to syslog-ng, 1) /dev/log forwarding 2) reading the journal files The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source. To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded. I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue. On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi Any news regarding this issue? Making a recap of the findings: * Using a Debian 10 buster with first release with 3.36.1; * After some time "system()" source logs are not getting written to the destinations; * The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations; * One the things I noticed is that the socket to the journal seems to vanish during the error situation: It seems that somehow syslog-ng in unable to read from linux journal. Have you ever experienced this problem? Do know what can be wrong with the system? root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> Thanks in advance, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Alexandre Santos Sent: 19 de maio de 2022 09:25 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi Szilard, There is no filter: source syslog_ng_src { internal(); }; destination d_localfile_syslog_ng { program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10" flags(syslog-protocol) suppress(5) disk-buffer( mem-buf-size(2097152) disk-buf-size(4194304) reliable(yes) dir("/tmp") ) ); }; log { source(syslog_ng_src); destination(d_localfile_syslog_ng); flags(flow-control); }; Thanks and Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Szilard Parrag (sparrag) Sent: 19 de maio de 2022 08:59 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=a2TZ40wm6cSogr9U8Cohka%2FM%2FJS25mF2oAKN5UqcyZA%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SB1BSl3EuYVAtN3Phj9wMWSv5jwjN0zJ%2BAb0ArWNYkg%3D&reserved=0> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=JG%2FfZBGTh6BuFq76QXUhXLpnrkxOVRhod9Cy3sRRhzU%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SldWSDsKiv7WXDjZMsYrUYiAMryqgnKX6zMqwVMN9C8%3D&reserved=0>
sorry, no I didn't. I just looked at your configs, If I see correctly then in the error situation the local logs are stalled, counters are not incremented. The journald "processed" counter stays at 85558, while immediately after reload it goes to 271377, which seems to indicate that for some reason we are not getting updates on the journals. I am reading the code to understand how we poll the journal and I hope I'll have some input how to debug this. Bazsi On Mon, Oct 31, 2022 at 5:15 PM Alexandre Santos <ASantos@infinera.com> wrote:
Hi,
Did you have a chance to analyze this?
Thanks & Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> * On Behalf Of *Alexandre Santos *Sent:* 16 de setembro de 2022 14:38 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Balazs,
Do you have any other idea of how to debug this?
From the last time I was able to reproduce the problem, I found the following:
*New developments, on this.*
*After another test this time with no remote destinations configured, the issue happened again:*
*In attachment I am sending:*
*- The stats in error condition: 42116.inerror.stats.txt, where ‘src.journald;s_src#0;journal;a;stamp;1651142246’*
*- The stats in error condition after some time: 42116.inerror.15m.stats.txt where ‘src.journald;s_src#0;journal;a;stamp;1651142246’ *
*- The stats after recovering the system with reload: 42116.after.stats.txt*
*- The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf*
*It seems the ‘src.journald;s_src#0;journal;a;stamp;1651142246’did not changed. Does this means that last read timestamp from journal did not changed?*
*Logs from the UDP source are still being written to the /var/log/… files.*
Cheers,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Balazs Scheidler *Sent:* 1 de julho de 2022 21:02 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us.
If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop.
Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume.
I don't really have any other ideas at the moment. Still on vacation. :)
On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos@infinera.com> wrote:
Hi Balazs,
Thanks for the feedback!
It is definitely using journald, as you can see below.
When you say “Try to remove the syslog-ng persist file and check if reading the journal restarts.”, this is to do when the system in the error condition, right?
Regards,
Alex
*source s_src {*
*#Start Block source generator system*
*channel {*
* source {*
*systemd-journal();*
* }; # source*
*channel {*
* channel {*
* parser {*
*#Start Block parser generator app-parser*
*channel {*
* junction {*
*channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };*
*}*
*#End Block parser generator app-parser*
*;*
* };*
* flags(final);*
* };*
* channel { flags(final); };*
*};*
*}; # channel*
*#End Block source generator system*
*;*
* internal();*
* syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));*
*};*
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Balazs Scheidler *Sent:* 26 de junho de 2022 06:37 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi,
I haven't seen anything like this. We are reading the journal files using libsystemd.
Try to remove the syslog-ng persist file and check if reading the journal restarts.
Also there's can be two ways of local messages getting to syslog-ng,
1) /dev/log forwarding
2) reading the journal files
The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.
To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.
I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.
On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com> wrote:
Hi
Any news regarding this issue?
Making a recap of the findings:
- Using a Debian 10 buster with first release with 3.36.1;
- After some time “system()” source logs are not getting written to the destinations; - The log messages from other sources, internal() and syslog(…) continue to work fine, being written to the destinations; - One the things I noticed is that the socket to the journal seems to vanish during the error situation:
It seems that somehow syslog-ng in unable to read from linux journal.
*Have you ever experienced this problem?*
*Do know what can be wrong with the system?*
root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
Thanks in advance,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Alexandre Santos *Sent:* 19 de maio de 2022 09:25 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Szilard,
There is no filter:
source syslog_ng_src {
internal();
};
destination d_localfile_syslog_ng {
program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
flags(syslog-protocol)
suppress(5)
disk-buffer(
mem-buf-size(2097152)
disk-buf-size(4194304)
reliable(yes)
dir("/tmp")
)
);
};
log {
source(syslog_ng_src);
destination(d_localfile_syslog_ng);
flags(flow-control);
};
Thanks and Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Szilard Parrag (sparrag) *Sent:* 19 de maio de 2022 08:59 *To:* syslog-ng@lists.balabit.hu *Subject:* Re: [syslog-ng] Local sources seem not to be working
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Alex,
We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.
Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?
Szilard
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=a2TZ40wm6cSogr9U8Cohka%2FM%2FJS25mF2oAKN5UqcyZA%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SB1BSl3EuYVAtN3Phj9wMWSv5jwjN0zJ%2BAb0ArWNYkg%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=JG%2FfZBGTh6BuFq76QXUhXLpnrkxOVRhod9Cy3sRRhzU%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SldWSDsKiv7WXDjZMsYrUYiAMryqgnKX6zMqwVMN9C8%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
I've occasionally seen problems, and am running into one now, where logs aren't being delivered and exhibiting similar behavior as described in this thread. systemd-journal seems to be a common theme, but in my cases I seem to run into problems when syslog-ng is running in an OpenVZ virtual container environment. Is that possibly the case here Alex? I have to do some more debugging myself and if I find any clues I'll report back in case we're experiencing something similar. What I've seen is almost like an apparmor issue, but I've not found any obvious connection to that yet. John
Ok, I've now read the relevant code to see how the journald polling operates and now have a general understanding. Couple of questions and notes. 1) which version of of systemd do you have? (Debian buster currently has 247.3-6~bpo10+1), is that right? 2) syslog-ng 3.36.1, right? 3) you are using flow-control + disk buffer 4) your system seems to be lightly loaded, you seem to have had ~30 messages/sec in the journal between stalling and recovering via SIGHUP (based on the "processed" counts and the last timestamp) syslog-ng uses the libsystemd library to poll for changes, which in turn uses inotify to watch the directories and files that make up the journal. You should see the inotify fd in the lsof output. I have mine like this: syslog-ng 3233 root 11r a_inode 0,14 0 10439 inotify You should also have a couple of "eventpoll" fd's open, in lsof output this should look like this: syslog-ng 3233 root 3u a_inode 0,14 0 10439 [eventpoll] syslog-ng 3233 root 13u a_inode 0,14 0 10439 [eventpoll] As you can see, I have two of these eventpoll fds, #3 is the main one, usually that comes first. If you do an strace on the syslog-ng process (command was "strace -s 256 -f -p <syslogng-pid> -o aaa"), you can see the epoll being manipulated: # we are grepping for all epoll operations with fd #3 $ grep 'epoll.*(3' aaa 3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260> 3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007> 3233 07:12:56.005265 epoll_wait(3, <unfinished ...> 625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008> 3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.005864 epoll_wait(3, <unfinished ...> 625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006> 3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790> 3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009> 3233 07:12:56.257518 epoll_wait(3, <unfinished ...> 625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021> 3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.258250 epoll_wait(3, <unfinished ...> 625546 07:12:56.258411 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000013> 3233 07:12:56.258496 epoll_wait(3, [{EPOLLIN, {u32=1319484552, u64=94426175487112}}], 10, -1) = 1 <0.203908> 3233 07:12:56.462711 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 1000) = 1 <0.123872> 3233 07:12:56.587406 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000015> 3233 07:12:56.587542 epoll_wait(3, <unfinished ...>
From those events, you can see that syslog-ng deletes and adds fd #11 (which is my inotify descriptor) and fd #4 (which is an event). If I zoom in to fd 11 in addition to epoll:
$ egrep '(\(11)|(epoll.*\(3)' aaa # first event and its reaction: 3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260> 3233 07:12:56.004800 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000019> 3233 07:12:56.005090 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000006> 3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007> 3233 07:12:56.005265 epoll_wait(3, <unfinished ...> 625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008> 3233 07:12:56.005595 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000007> 3233 07:12:56.005618 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000006> 3233 07:12:56.005659 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000007> 3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.005864 epoll_wait(3, <unfinished ...> # second event and its reaction: 625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006> 3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790> 3233 07:12:56.256994 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000026> 3233 07:12:56.257416 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000008> 3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009> 3233 07:12:56.257518 epoll_wait(3, <unfinished ...> 625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021> 3233 07:12:56.257940 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000011> 3233 07:12:56.257978 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000009> 3233 07:12:56.258017 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000011> 3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.258250 epoll_wait(3, <unfinished ...> please notice that syslog-ng reads fd #11 right after epoll indicates there's an event available. The read() has a payload which can be decoded according to struct inotify_event. In both cases, after the thread "625546" starts executing, we use the journald APIs to fetch the message. It's not v ery visible from the strace as sdjournal seems to access the files via mmap or something, but you should definitely see the disk buffer being written as messages get delivered, since you are using a reliable disk buffer (which you may not need btw, but that's a separate discussion). The reasons we might stall could be: - the file is not modified at all (this could be ruled out) - we don't poll the inotify fd (you can confirm this via strace as above) - we do poll the fd, but we don't read the events from it (again strace helps here) - we do poll fd fd and read the events but somehow we don't respond to it properly (this is probably a bug) - our worker wakes up in response to the event but then it doesn't consume messages from the journal (this is probably a bug) If we are not polling the inotify fd, that means that the systemd-journal() source might be suspended. This can happen for example because of flow control: - a source is suspended if some of its flow-controlled destinations are unable to consume the messages it produces. in your case, you have a disk buffer and that consumes all messages up to its capacity. I didn't see disk buffers that were full (the queued counters are all zero), you were also saying that those same files are being written, messages delivered normally. - if for some reason we are stuck in the "suspended" state and the wakeup mechanism does not work (this is probably a bug) With all this said, can you please confirm that we are polling the inotify fd from our main thread and that you can see the inotify event being consumed? Or you don't see that all. Of course while syslog-ng is working, this should operate normally. The trick is to look at it when it is stalled. Thanks and again sorry for this to take so long. On Tue, Nov 1, 2022 at 6:36 AM Balazs Scheidler <bazsi77@gmail.com> wrote:
sorry, no I didn't.
I just looked at your configs, If I see correctly then in the error situation the local logs are stalled, counters are not incremented. The journald "processed" counter stays at 85558, while immediately after reload it goes to 271377, which seems to indicate that for some reason we are not getting updates on the journals.
I am reading the code to understand how we poll the journal and I hope I'll have some input how to debug this.
Bazsi
On Mon, Oct 31, 2022 at 5:15 PM Alexandre Santos <ASantos@infinera.com> wrote:
Hi,
Did you have a chance to analyze this?
Thanks & Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> * On Behalf Of *Alexandre Santos *Sent:* 16 de setembro de 2022 14:38 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Balazs,
Do you have any other idea of how to debug this?
From the last time I was able to reproduce the problem, I found the following:
*New developments, on this.*
*After another test this time with no remote destinations configured, the issue happened again:*
*In attachment I am sending:*
*- The stats in error condition: 42116.inerror.stats.txt, where ‘src.journald;s_src#0;journal;a;stamp;1651142246’*
*- The stats in error condition after some time: 42116.inerror.15m.stats.txt where ‘src.journald;s_src#0;journal;a;stamp;1651142246’ *
*- The stats after recovering the system with reload: 42116.after.stats.txt*
*- The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf*
*It seems the ‘src.journald;s_src#0;journal;a;stamp;1651142246’did not changed. Does this means that last read timestamp from journal did not changed?*
*Logs from the UDP source are still being written to the /var/log/… files.*
Cheers,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Balazs Scheidler *Sent:* 1 de julho de 2022 21:02 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us.
If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop.
Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume.
I don't really have any other ideas at the moment. Still on vacation. :)
On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos@infinera.com> wrote:
Hi Balazs,
Thanks for the feedback!
It is definitely using journald, as you can see below.
When you say “Try to remove the syslog-ng persist file and check if reading the journal restarts.”, this is to do when the system in the error condition, right?
Regards,
Alex
*source s_src {*
*#Start Block source generator system*
*channel {*
* source {*
*systemd-journal();*
* }; # source*
*channel {*
* channel {*
* parser {*
*#Start Block parser generator app-parser*
*channel {*
* junction {*
*channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };*
*}*
*#End Block parser generator app-parser*
*;*
* };*
* flags(final);*
* };*
* channel { flags(final); };*
*};*
*}; # channel*
*#End Block source generator system*
*;*
* internal();*
* syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));*
*};*
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Balazs Scheidler *Sent:* 26 de junho de 2022 06:37 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi,
I haven't seen anything like this. We are reading the journal files using libsystemd.
Try to remove the syslog-ng persist file and check if reading the journal restarts.
Also there's can be two ways of local messages getting to syslog-ng,
1) /dev/log forwarding
2) reading the journal files
The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.
To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.
I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.
On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com> wrote:
Hi
Any news regarding this issue?
Making a recap of the findings:
- Using a Debian 10 buster with first release with 3.36.1;
- After some time “system()” source logs are not getting written to the destinations; - The log messages from other sources, internal() and syslog(…) continue to work fine, being written to the destinations; - One the things I noticed is that the socket to the journal seems to vanish during the error situation:
It seems that somehow syslog-ng in unable to read from linux journal.
*Have you ever experienced this problem?*
*Do know what can be wrong with the system?*
root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
Thanks in advance,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Alexandre Santos *Sent:* 19 de maio de 2022 09:25 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Szilard,
There is no filter:
source syslog_ng_src {
internal();
};
destination d_localfile_syslog_ng {
program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
flags(syslog-protocol)
suppress(5)
disk-buffer(
mem-buf-size(2097152)
disk-buf-size(4194304)
reliable(yes)
dir("/tmp")
)
);
};
log {
source(syslog_ng_src);
destination(d_localfile_syslog_ng);
flags(flow-control);
};
Thanks and Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Szilard Parrag (sparrag) *Sent:* 19 de maio de 2022 08:59 *To:* syslog-ng@lists.balabit.hu *Subject:* Re: [syslog-ng] Local sources seem not to be working
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Alex,
We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.
Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?
Szilard
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=a2TZ40wm6cSogr9U8Cohka%2FM%2FJS25mF2oAKN5UqcyZA%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SB1BSl3EuYVAtN3Phj9wMWSv5jwjN0zJ%2BAb0ArWNYkg%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=JG%2FfZBGTh6BuFq76QXUhXLpnrkxOVRhod9Cy3sRRhzU%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SldWSDsKiv7WXDjZMsYrUYiAMryqgnKX6zMqwVMN9C8%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
-- Bazsi
From the last tests with info that was provided regarding journald polling, it seems that the system is hitting this problem: https://github.com/syslog-ng/syslog-ng/issues/2836
Still, the last time I catch the problem, I have the following: root@machine:/ # cat /proc/sys/kernel/random/boot_id e473e508-6bce-412a-b7b8-ff44b5dbea9b root@machine:/ # journalctl --list-boots -1 8f5414009b064ef68186e2b91a9f251d Tue 2023-01-10 02:41:58 CST-Tue 2023-01-10 02:42:32 CST 0 a74228741b0240ba93621f15bd58b4c4 Tue 2023-01-10 02:57:57 CST-Wed 2023-01-11 19:39:20 CST The boot id does not match any of the journal boots. So restart of syslog-ng does not clear the problem. Do you know how can this happen? Regarding your questions: 1) systemd 241 (241) 2) syslog-ng 3.36.1 3) Yes 4) Yes, it is quite loaded. Thanks, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Balazs Scheidler Sent: 1 de novembro de 2022 06:49 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working Ok, I've now read the relevant code to see how the journald polling operates and now have a general understanding. Couple of questions and notes. 1) which version of of systemd do you have? (Debian buster currently has 247.3-6~bpo10+1), is that right? 2) syslog-ng 3.36.1, right? 3) you are using flow-control + disk buffer 4) your system seems to be lightly loaded, you seem to have had ~30 messages/sec in the journal between stalling and recovering via SIGHUP (based on the "processed" counts and the last timestamp) syslog-ng uses the libsystemd library to poll for changes, which in turn uses inotify to watch the directories and files that make up the journal. You should see the inotify fd in the lsof output. I have mine like this: syslog-ng 3233 root 11r a_inode 0,14 0 10439 inotify You should also have a couple of "eventpoll" fd's open, in lsof output this should look like this: syslog-ng 3233 root 3u a_inode 0,14 0 10439 [eventpoll] syslog-ng 3233 root 13u a_inode 0,14 0 10439 [eventpoll] As you can see, I have two of these eventpoll fds, #3 is the main one, usually that comes first. If you do an strace on the syslog-ng process (command was "strace -s 256 -f -p <syslogng-pid> -o aaa"), you can see the epoll being manipulated: # we are grepping for all epoll operations with fd #3 $ grep 'epoll.*(3' aaa 3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260> 3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007> 3233 07:12:56.005265 epoll_wait(3, <unfinished ...> 625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008> 3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.005864 epoll_wait(3, <unfinished ...> 625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006> 3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790> 3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009> 3233 07:12:56.257518 epoll_wait(3, <unfinished ...> 625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021> 3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.258250 epoll_wait(3, <unfinished ...> 625546 07:12:56.258411 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000013> 3233 07:12:56.258496 epoll_wait(3, [{EPOLLIN, {u32=1319484552, u64=94426175487112}}], 10, -1) = 1 <0.203908> 3233 07:12:56.462711 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 1000) = 1 <0.123872> 3233 07:12:56.587406 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000015> 3233 07:12:56.587542 epoll_wait(3, <unfinished ...>
From those events, you can see that syslog-ng deletes and adds fd #11 (which is my inotify descriptor) and fd #4 (which is an event). If I zoom in to fd 11 in addition to epoll:
$ egrep '(\(11)|(epoll.*\(3)' aaa # first event and its reaction: 3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260> 3233 07:12:56.004800 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000019> 3233 07:12:56.005090 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000006> 3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007> 3233 07:12:56.005265 epoll_wait(3, <unfinished ...> 625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008> 3233 07:12:56.005595 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000007> 3233 07:12:56.005618 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000006> 3233 07:12:56.005659 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000007> 3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.005864 epoll_wait(3, <unfinished ...> # second event and its reaction: 625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006> 3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790> 3233 07:12:56.256994 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000026> 3233 07:12:56.257416 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000008> 3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009> 3233 07:12:56.257518 epoll_wait(3, <unfinished ...> 625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021> 3233 07:12:56.257940 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000011> 3233 07:12:56.257978 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000009> 3233 07:12:56.258017 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000011> 3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.258250 epoll_wait(3, <unfinished ...> please notice that syslog-ng reads fd #11 right after epoll indicates there's an event available. The read() has a payload which can be decoded according to struct inotify_event. In both cases, after the thread "625546" starts executing, we use the journald APIs to fetch the message. It's not v ery visible from the strace as sdjournal seems to access the files via mmap or something, but you should definitely see the disk buffer being written as messages get delivered, since you are using a reliable disk buffer (which you may not need btw, but that's a separate discussion). The reasons we might stall could be: * the file is not modified at all (this could be ruled out) * we don't poll the inotify fd (you can confirm this via strace as above) * we do poll the fd, but we don't read the events from it (again strace helps here) * we do poll fd fd and read the events but somehow we don't respond to it properly (this is probably a bug) * our worker wakes up in response to the event but then it doesn't consume messages from the journal (this is probably a bug) If we are not polling the inotify fd, that means that the systemd-journal() source might be suspended. This can happen for example because of flow control: * a source is suspended if some of its flow-controlled destinations are unable to consume the messages it produces. in your case, you have a disk buffer and that consumes all messages up to its capacity. I didn't see disk buffers that were full (the queued counters are all zero), you were also saying that those same files are being written, messages delivered normally. * if for some reason we are stuck in the "suspended" state and the wakeup mechanism does not work (this is probably a bug) With all this said, can you please confirm that we are polling the inotify fd from our main thread and that you can see the inotify event being consumed? Or you don't see that all. Of course while syslog-ng is working, this should operate normally. The trick is to look at it when it is stalled. Thanks and again sorry for this to take so long. On Tue, Nov 1, 2022 at 6:36 AM Balazs Scheidler <bazsi77@gmail.com<mailto:bazsi77@gmail.com>> wrote: sorry, no I didn't. I just looked at your configs, If I see correctly then in the error situation the local logs are stalled, counters are not incremented. The journald "processed" counter stays at 85558, while immediately after reload it goes to 271377, which seems to indicate that for some reason we are not getting updates on the journals. I am reading the code to understand how we poll the journal and I hope I'll have some input how to debug this. Bazsi On Mon, Oct 31, 2022 at 5:15 PM Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi, Did you have a chance to analyze this? Thanks & Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Alexandre Santos Sent: 16 de setembro de 2022 14:38 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi Balazs, Do you have any other idea of how to debug this?
From the last time I was able to reproduce the problem, I found the following:
New developments, on this. After another test this time with no remote destinations configured, the issue happened again: In attachment I am sending: - The stats in error condition: 42116.inerror.stats.txt, where 'src.journald;s_src#0;journal;a;stamp;1651142246' - The stats in error condition after some time: 42116.inerror.15m.stats.txt where 'src.journald;s_src#0;journal;a;stamp;1651142246' - The stats after recovering the system with reload: 42116.after.stats.txt - The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf It seems the 'src.journald;s_src#0;journal;a;stamp;1651142246'did not changed. Does this means that last read timestamp from journal did not changed? Logs from the UDP source are still being written to the /var/log/... files. Cheers, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Balazs Scheidler Sent: 1 de julho de 2022 21:02 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us. If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop. Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume. I don't really have any other ideas at the moment. Still on vacation. :) On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi Balazs, Thanks for the feedback! It is definitely using journald, as you can see below. When you say "Try to remove the syslog-ng persist file and check if reading the journal restarts.", this is to do when the system in the error condition, right? Regards, Alex source s_src { #Start Block source generator system channel { source { systemd-journal(); }; # source channel { channel { parser { #Start Block parser generator app-parser channel { junction { channel { filter { tags('.app.doesnotexist'); }; flags(final); }; }; } #End Block parser generator app-parser ; }; flags(final); }; channel { flags(final); }; }; }; # channel #End Block source generator system ; internal(); syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no)); }; From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Balazs Scheidler Sent: 26 de junho de 2022 06:37 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi, I haven't seen anything like this. We are reading the journal files using libsystemd. Try to remove the syslog-ng persist file and check if reading the journal restarts. Also there's can be two ways of local messages getting to syslog-ng, 1) /dev/log forwarding 2) reading the journal files The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source. To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded. I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue. On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com<mailto:ASantos@infinera.com>> wrote: Hi Any news regarding this issue? Making a recap of the findings: * Using a Debian 10 buster with first release with 3.36.1; * After some time "system()" source logs are not getting written to the destinations; * The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations; * One the things I noticed is that the socket to the journal seems to vanish during the error situation: It seems that somehow syslog-ng in unable to read from linux journal. Have you ever experienced this problem? Do know what can be wrong with the system? root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal> Thanks in advance, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Alexandre Santos Sent: 19 de maio de 2022 09:25 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Local sources seem not to be working Hi Szilard, There is no filter: source syslog_ng_src { internal(); }; destination d_localfile_syslog_ng { program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10" flags(syslog-protocol) suppress(5) disk-buffer( mem-buf-size(2097152) disk-buf-size(4194304) reliable(yes) dir("/tmp") ) ); }; log { source(syslog_ng_src); destination(d_localfile_syslog_ng); flags(flow-control); }; Thanks and Regards, Alex From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Szilard Parrag (sparrag) Sent: 19 de maio de 2022 08:59 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Local sources seem not to be working CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hx4vnC4ovnJHzP%2FOvBhwD7JynP2dgYKQg6e%2BeK7xEaY%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jiIrysNzV3agMWDR%2FwQn%2FZ7WvyXYmUODE%2BcsF8MYzCM%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=epn0Ap2nNzhsEahBAh2JNwLerEIaHKab6gweveZ8l1Q%3D&reserved=0> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hx4vnC4ovnJHzP%2FOvBhwD7JynP2dgYKQg6e%2BeK7xEaY%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jiIrysNzV3agMWDR%2FwQn%2FZ7WvyXYmUODE%2BcsF8MYzCM%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=epn0Ap2nNzhsEahBAh2JNwLerEIaHKab6gweveZ8l1Q%3D&reserved=0> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hx4vnC4ovnJHzP%2FOvBhwD7JynP2dgYKQg6e%2BeK7xEaY%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jiIrysNzV3agMWDR%2FwQn%2FZ7WvyXYmUODE%2BcsF8MYzCM%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=epn0Ap2nNzhsEahBAh2JNwLerEIaHKab6gweveZ8l1Q%3D&reserved=0> -- Bazsi -- Bazsi
The /proc based file does not have the same format as systemd uses it although it should be the same bytes, just formatted differently. I tried to creatively match bytes from your boot id with the systemd entries (like byte ordering mismatch, etc) but they indeed look like different values. If you look at the logs using "journalctl -fb" does that display log messages? Obviously this shouldn't happen. I have a work in progress PR to resolve the 'time-goes-backward' problem, here. https://github.com/syslog-ng/syslog-ng/pull/4245 Got distracted with various stuff ATM, but testing/feedback helps there too. Also, as a workaround, journald can be bypassed if you don't need the journals themselves, that gets rid off a lot of complexity. This can be done by forcing /dev/log to be processed by syslog-ng instead of the journald. One probably needs to change the socket activation unit for journald, and tell syslog-ng to use /dev/log even in light of running a systemd based setup. On Wed, Jan 11, 2023, 00:37 Alexandre Santos <ASantos@infinera.com> wrote:
From the last tests with info that was provided regarding journald polling, it seems that the system is hitting this problem:
https://github.com/syslog-ng/syslog-ng/issues/2836
Still, the last time I catch the problem, I have the following:
root@machine:/ # cat /proc/sys/kernel/random/boot_id
e473e508-6bce-412a-b7b8-ff44b5dbea9b
root@machine:/ # journalctl --list-boots
-1 8f5414009b064ef68186e2b91a9f251d Tue 2023-01-10 02:41:58 CST—Tue 2023-01-10 02:42:32 CST
0 a74228741b0240ba93621f15bd58b4c4 Tue 2023-01-10 02:57:57 CST—Wed 2023-01-11 19:39:20 CST
The boot id does not match any of the journal boots. So restart of syslog-ng does not clear the problem.
*Do you know how can this happen?*
Regarding your questions:
1) systemd 241 (241)
2) syslog-ng 3.36.1
3) Yes
4) Yes, it is quite loaded.
Thanks, Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> * On Behalf Of *Balazs Scheidler *Sent:* 1 de novembro de 2022 06:49 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Ok, I've now read the relevant code to see how the journald polling operates and now have a general understanding.
Couple of questions and notes.
1) which version of of systemd do you have? (Debian buster currently has 247.3-6~bpo10+1), is that right?
2) syslog-ng 3.36.1, right?
3) you are using flow-control + disk buffer
4) your system seems to be lightly loaded, you seem to have had ~30 messages/sec in the journal between stalling and recovering via SIGHUP (based on the "processed" counts and the last timestamp)
syslog-ng uses the libsystemd library to poll for changes, which in turn uses inotify to watch the directories and files that make up the journal. You should see the inotify fd in the lsof output. I have mine like this:
syslog-ng 3233 root 11r a_inode 0,14 0 10439 inotify
You should also have a couple of "eventpoll" fd's open, in lsof output this should look like this:
syslog-ng 3233 root 3u a_inode 0,14 0 10439 [eventpoll] syslog-ng 3233 root 13u a_inode 0,14 0 10439 [eventpoll]
As you can see, I have two of these eventpoll fds, #3 is the main one, usually that comes first.
If you do an strace on the syslog-ng process (command was "strace -s 256 -f -p <syslogng-pid> -o aaa"), you can see the epoll being manipulated:
# we are grepping for all epoll operations with fd #3
$ grep 'epoll.*(3' aaa 3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260> 3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007> 3233 07:12:56.005265 epoll_wait(3, <unfinished ...> 625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008> 3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.005864 epoll_wait(3, <unfinished ...> 625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006> 3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790> 3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009> 3233 07:12:56.257518 epoll_wait(3, <unfinished ...> 625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021> 3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.258250 epoll_wait(3, <unfinished ...> 625546 07:12:56.258411 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000013> 3233 07:12:56.258496 epoll_wait(3, [{EPOLLIN, {u32=1319484552, u64=94426175487112}}], 10, -1) = 1 <0.203908> 3233 07:12:56.462711 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 1000) = 1 <0.123872> 3233 07:12:56.587406 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000015> 3233 07:12:56.587542 epoll_wait(3, <unfinished ...>
From those events, you can see that syslog-ng deletes and adds fd #11 (which is my inotify descriptor) and fd #4 (which is an event). If I zoom in to fd 11 in addition to epoll:
$ egrep '(\(11)|(epoll.*\(3)' aaa
# first event and its reaction:
3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260> 3233 07:12:56.004800 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000019> 3233 07:12:56.005090 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000006> 3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007> 3233 07:12:56.005265 epoll_wait(3, <unfinished ...> 625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008> 3233 07:12:56.005595 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000007> 3233 07:12:56.005618 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000006> 3233 07:12:56.005659 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000007> 3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.005864 epoll_wait(3, <unfinished ...>
# second event and its reaction:
625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006> 3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790> 3233 07:12:56.256994 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000026> 3233 07:12:56.257416 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000008> 3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009> 3233 07:12:56.257518 epoll_wait(3, <unfinished ...> 625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021> 3233 07:12:56.257940 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000011> 3233 07:12:56.257978 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000009> 3233 07:12:56.258017 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000011> 3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...> 3233 07:12:56.258250 epoll_wait(3, <unfinished ...>
please notice that syslog-ng reads fd #11 right after epoll indicates there's an event available. The read() has a payload which can be decoded according to struct inotify_event.
In both cases, after the thread "625546" starts executing, we use the journald APIs to fetch the message. It's not v ery visible from the strace as sdjournal seems to access the files via mmap or something, but you should definitely see the disk buffer being written as messages get delivered, since you are using a reliable disk buffer (which you may not need btw, but that's a separate discussion).
The reasons we might stall could be:
- the file is not modified at all (this could be ruled out) - we don't poll the inotify fd (you can confirm this via strace as above) - we do poll the fd, but we don't read the events from it (again strace helps here) - we do poll fd fd and read the events but somehow we don't respond to it properly (this is probably a bug) - our worker wakes up in response to the event but then it doesn't consume messages from the journal (this is probably a bug)
If we are not polling the inotify fd, that means that the systemd-journal() source might be suspended. This can happen for example because of flow control:
- a source is suspended if some of its flow-controlled destinations are unable to consume the messages it produces. in your case, you have a disk buffer and that consumes all messages up to its capacity. I didn't see disk buffers that were full (the queued counters are all zero), you were also saying that those same files are being written, messages delivered normally. - if for some reason we are stuck in the "suspended" state and the wakeup mechanism does not work (this is probably a bug)
With all this said, can you please confirm that we are polling the inotify fd from our main thread and that you can see the inotify event being consumed? Or you don't see that all.
Of course while syslog-ng is working, this should operate normally. The trick is to look at it when it is stalled.
Thanks and again sorry for this to take so long.
On Tue, Nov 1, 2022 at 6:36 AM Balazs Scheidler <bazsi77@gmail.com> wrote:
sorry, no I didn't.
I just looked at your configs, If I see correctly then in the error situation the local logs are stalled, counters are not incremented. The journald "processed" counter stays at 85558, while immediately after reload it goes to 271377, which seems to indicate that for some reason we are not getting updates on the journals.
I am reading the code to understand how we poll the journal and I hope I'll have some input how to debug this.
Bazsi
On Mon, Oct 31, 2022 at 5:15 PM Alexandre Santos <ASantos@infinera.com> wrote:
Hi,
Did you have a chance to analyze this?
Thanks & Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Alexandre Santos *Sent:* 16 de setembro de 2022 14:38 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Balazs,
Do you have any other idea of how to debug this?
From the last time I was able to reproduce the problem, I found the following:
*New developments, on this.*
*After another test this time with no remote destinations configured, the issue happened again:*
*In attachment I am sending:*
*- The stats in error condition: 42116.inerror.stats.txt, where ‘src.journald;s_src#0;journal;a;stamp;1651142246’*
*- The stats in error condition after some time: 42116.inerror.15m.stats.txt where ‘src.journald;s_src#0;journal;a;stamp;1651142246’ *
*- The stats after recovering the system with reload: 42116.after.stats.txt*
*- The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf*
*It seems the ‘src.journald;s_src#0;journal;a;stamp;1651142246’did not changed. Does this means that last read timestamp from journal did not changed?*
*Logs from the UDP source are still being written to the /var/log/… files.*
Cheers,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Balazs Scheidler *Sent:* 1 de julho de 2022 21:02 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us.
If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop.
Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume.
I don't really have any other ideas at the moment. Still on vacation. :)
On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos@infinera.com> wrote:
Hi Balazs,
Thanks for the feedback!
It is definitely using journald, as you can see below.
When you say “Try to remove the syslog-ng persist file and check if reading the journal restarts.”, this is to do when the system in the error condition, right?
Regards,
Alex
*source s_src {*
*#Start Block source generator system*
*channel {*
* source {*
*systemd-journal();*
* }; # source*
*channel {*
* channel {*
* parser {*
*#Start Block parser generator app-parser*
*channel {*
* junction {*
*channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };*
*}*
*#End Block parser generator app-parser*
*;*
* };*
* flags(final);*
* };*
* channel { flags(final); };*
*};*
*}; # channel*
*#End Block source generator system*
*;*
* internal();*
* syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));*
*};*
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Balazs Scheidler *Sent:* 26 de junho de 2022 06:37 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi,
I haven't seen anything like this. We are reading the journal files using libsystemd.
Try to remove the syslog-ng persist file and check if reading the journal restarts.
Also there's can be two ways of local messages getting to syslog-ng,
1) /dev/log forwarding
2) reading the journal files
The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.
To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.
I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.
On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos@infinera.com> wrote:
Hi
Any news regarding this issue?
Making a recap of the findings:
- Using a Debian 10 buster with first release with 3.36.1;
- After some time “system()” source logs are not getting written to the destinations; - The log messages from other sources, internal() and syslog(…) continue to work fine, being written to the destinations; - One the things I noticed is that the socket to the journal seems to vanish during the error situation:
It seems that somehow syslog-ng in unable to read from linux journal.
*Have you ever experienced this problem?*
*Do know what can be wrong with the system?*
root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
Thanks in advance,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Alexandre Santos *Sent:* 19 de maio de 2022 09:25 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Local sources seem not to be working
Hi Szilard,
There is no filter:
source syslog_ng_src {
internal();
};
destination d_localfile_syslog_ng {
program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
flags(syslog-protocol)
suppress(5)
disk-buffer(
mem-buf-size(2097152)
disk-buf-size(4194304)
reliable(yes)
dir("/tmp")
)
);
};
log {
source(syslog_ng_src);
destination(d_localfile_syslog_ng);
flags(flow-control);
};
Thanks and Regards,
Alex
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Szilard Parrag (sparrag) *Sent:* 19 de maio de 2022 08:59 *To:* syslog-ng@lists.balabit.hu *Subject:* Re: [syslog-ng] Local sources seem not to be working
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Alex,
We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.
Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?
Szilard
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hx4vnC4ovnJHzP%2FOvBhwD7JynP2dgYKQg6e%2BeK7xEaY%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jiIrysNzV3agMWDR%2FwQn%2FZ7WvyXYmUODE%2BcsF8MYzCM%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=epn0Ap2nNzhsEahBAh2JNwLerEIaHKab6gweveZ8l1Q%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hx4vnC4ovnJHzP%2FOvBhwD7JynP2dgYKQg6e%2BeK7xEaY%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jiIrysNzV3agMWDR%2FwQn%2FZ7WvyXYmUODE%2BcsF8MYzCM%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=epn0Ap2nNzhsEahBAh2JNwLerEIaHKab6gweveZ8l1Q%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hx4vnC4ovnJHzP%2FOvBhwD7JynP2dgYKQg6e%2BeK7xEaY%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jiIrysNzV3agMWDR%2FwQn%2FZ7WvyXYmUODE%2BcsF8MYzCM%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C14e11b4e57564fdd257808dabbd54209%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C638028821866307428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=epn0Ap2nNzhsEahBAh2JNwLerEIaHKab6gweveZ8l1Q%3D&reserved=0>
--
Bazsi
--
Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Alexandre Santos
-
Balazs Scheidler
-
John Kristoff
-
Szilard Parrag (sparrag)