[syslog-ng] Local sources seem not to be working
Balazs Scheidler
bazsi77 at gmail.com
Sun Jun 26 05:36:38 UTC 2022
Hi,
I haven't seen anything like this. We are reading the journal files using
libsystemd.
Try to remove the syslog-ng persist file and check if reading the journal
restarts.
Also there's can be two ways of local messages getting to syslog-ng,
1) /dev/log forwarding
2) reading the journal files
The first one is actively done by journald. Which one syslog-ng uses is
automatically detected by our system() source.
To see which one syslog-ng is trying to use, try to run it with
--preprocess-into=some-file and check how system() source is expanded.
I am unable to check the source code at the moment, so this is all from the
top-of-my-head, but I hope this already helps to troubleshoot the issue.
On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos at infinera.com> wrote:
> Hi
>
>
>
> Any news regarding this issue?
>
>
>
> Making a recap of the findings:
>
>
>
> - Using a Debian 10 buster with first release with 3.36.1;
>
>
> - After some time “system()” source logs are not getting written to
> the destinations;
> - The log messages from other sources, internal() and syslog(…)
> continue to work fine, being written to the destinations;
> - One the things I noticed is that the socket to the journal seems to
> vanish during the error situation:
>
>
>
> It seems that somehow syslog-ng in unable to read from linux journal.
>
> *Have you ever experienced this problem?*
>
> *Do know what can be wrong with the system?*
>
>
>
>
>
> root at machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>
> systemd-j 1723 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 16u REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 24u REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> syslog-ng 3201 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root 14r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root 15r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 5r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 6r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> root@ machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>
> systemd-j 1723 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 16u REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 24u REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 5r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 6r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
>
>
> Thanks in advance,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> * On Behalf Of *Alexandre
> Santos
> *Sent:* 19 de maio de 2022 09:25
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> Hi Szilard,
>
>
>
> There is no filter:
>
>
>
> source syslog_ng_src {
>
> internal();
>
> };
>
>
>
> destination d_localfile_syslog_ng {
>
> program("/opt/machine/local/bin/write_with_rotation.sh
> /var/log/syslog-ng-internal.log 10 10"
>
> flags(syslog-protocol)
>
> suppress(5)
>
> disk-buffer(
>
> mem-buf-size(2097152)
>
> disk-buf-size(4194304)
>
> reliable(yes)
>
> dir("/tmp")
>
> )
>
> );
>
> };
>
> log {
>
> source(syslog_ng_src);
>
> destination(d_localfile_syslog_ng);
>
> flags(flow-control);
>
> };
>
>
>
> Thanks and Regards,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> * On Behalf Of *Szilard
> Parrag (sparrag)
> *Sent:* 19 de maio de 2022 08:59
> *To:* syslog-ng at lists.balabit.hu
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Alex,
>
>
>
> We've checked it too and syslog-ng does not release the file descriptor of
> journald even with flow-control enabled.
>
>
>
> Also, your internal logs seem rather terse, maybe there is a filter which
> filters out the important parts. Could you please check it?
>
>
>
> Szilard
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220626/d5ce66bd/attachment.htm>
More information about the syslog-ng
mailing list