[syslog-ng] Local sources seem not to be working
Alexandre Santos
ASantos at infinera.com
Thu Jun 30 17:43:11 UTC 2022
Hi Balazs,
Thanks for the feedback!
It is definitely using journald, as you can see below.
When you say "Try to remove the syslog-ng persist file and check if reading the journal restarts.", this is to do when the system in the error condition, right?
Regards,
Alex
source s_src {
#Start Block source generator system
channel {
source {
systemd-journal();
}; # source
channel {
channel {
parser {
#Start Block parser generator app-parser
channel {
junction {
channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };
}
#End Block parser generator app-parser
;
};
flags(final);
};
channel { flags(final); };
};
}; # channel
#End Block source generator system
;
internal();
syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));
};
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Balazs Scheidler
Sent: 26 de junho de 2022 06:37
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Local sources seem not to be working
Hi,
I haven't seen anything like this. We are reading the journal files using libsystemd.
Try to remove the syslog-ng persist file and check if reading the journal restarts.
Also there's can be two ways of local messages getting to syslog-ng,
1) /dev/log forwarding
2) reading the journal files
The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.
To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.
I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.
On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>> wrote:
Hi
Any news regarding this issue?
Making a recap of the findings:
* Using a Debian 10 buster with first release with 3.36.1;
* After some time "system()" source logs are not getting written to the destinations;
* The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations;
* One the things I noticed is that the socket to the journal seems to vanish during the error situation:
It seems that somehow syslog-ng in unable to read from linux journal.
Have you ever experienced this problem?
Do know what can be wrong with the system?
root at machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
Thanks in advance,
Alex
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Alexandre Santos
Sent: 19 de maio de 2022 09:25
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] Local sources seem not to be working
Hi Szilard,
There is no filter:
source syslog_ng_src {
internal();
};
destination d_localfile_syslog_ng {
program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
flags(syslog-protocol)
suppress(5)
disk-buffer(
mem-buf-size(2097152)
disk-buf-size(4194304)
reliable(yes)
dir("/tmp")
)
);
};
log {
source(syslog_ng_src);
destination(d_localfile_syslog_ng);
flags(flow-control);
};
Thanks and Regards,
Alex
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Szilard Parrag (sparrag)
Sent: 19 de maio de 2022 08:59
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Local sources seem not to be working
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Alex,
We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.
Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?
Szilard
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=WNrM7LVg0OQZ4H%2FdhbZd4mCQ%2F6dAUsu9gJlgEa2aoCk%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=O5s9C1DJkKe%2F%2FJRWzDpRO4tpU1I31b%2F%2BG1SBXmG26Ps%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=CiurkG9FrrVfxETCA8L5nVWoHofbzNrwn%2BP%2BpfAHX1k%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220630/2c49461b/attachment-0001.htm>
More information about the syslog-ng
mailing list