[syslog-ng] Local sources seem not to be working

[Alexandre Santos]

Hi

Any news regarding this issue?

Making a recap of the findings:


  *   Using a Debian 10 buster with first release with 3.36.1;

  *   After some time "system()" source logs are not getting written to the destinations;
  *   The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations;
  *   One the things I noticed is that the socket to the journal seems to vanish during the error situation:

It seems that somehow syslog-ng in unable to read from linux journal.
Have you ever experienced this problem?
Do know what can be wrong with the system?


root at machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
systemd-j 1723 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   16u   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   24u   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
syslog-ng 3201 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root   14r   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root   15r   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    5r   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    6r   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
systemd-j 1723 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   16u   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   24u   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    5r   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    6r   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal

Thanks in advance,
Alex

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Alexandre Santos
Sent: 19 de maio de 2022 09:25
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Local sources seem not to be working

Hi Szilard,

There is no filter:

source syslog_ng_src {
    internal();
};

destination d_localfile_syslog_ng {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
log {
    source(syslog_ng_src);
    destination(d_localfile_syslog_ng);
    flags(flow-control);
};

Thanks and Regards,
Alex

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Szilard Parrag (sparrag)
Sent: 19 de maio de 2022 08:59
To: syslog-ng at lists.balabit.hu
Subject: Re: [syslog-ng] Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Alex,

We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.

Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?

Szilard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220624/bde851f2/attachment-0001.htm>