[syslog-ng] Local sources seem not to be working
Alexandre Santos
ASantos at infinera.com
Mon Oct 31 16:15:14 UTC 2022
Hi,
Did you have a chance to analyze this?
Thanks & Regards,
Alex
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Alexandre Santos
Sent: 16 de setembro de 2022 14:38
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Local sources seem not to be working
Hi Balazs,
Do you have any other idea of how to debug this?
>From the last time I was able to reproduce the problem, I found the following:
New developments, on this.
After another test this time with no remote destinations configured, the issue happened again:
In attachment I am sending:
- The stats in error condition: 42116.inerror.stats.txt, where 'src.journald;s_src#0;journal;a;stamp;1651142246'
- The stats in error condition after some time: 42116.inerror.15m.stats.txt where 'src.journald;s_src#0;journal;a;stamp;1651142246'
- The stats after recovering the system with reload: 42116.after.stats.txt
- The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf
It seems the 'src.journald;s_src#0;journal;a;stamp;1651142246'did not changed. Does this means that last read timestamp from journal did not changed?
Logs from the UDP source are still being written to the /var/log/... files.
Cheers,
Alex
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Balazs Scheidler
Sent: 1 de julho de 2022 21:02
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] Local sources seem not to be working
This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us.
If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop.
Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming in volume.
I don't really have any other ideas at the moment. Still on vacation. :)
On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>> wrote:
Hi Balazs,
Thanks for the feedback!
It is definitely using journald, as you can see below.
When you say "Try to remove the syslog-ng persist file and check if reading the journal restarts.", this is to do when the system in the error condition, right?
Regards,
Alex
source s_src {
#Start Block source generator system
channel {
source {
systemd-journal();
}; # source
channel {
channel {
parser {
#Start Block parser generator app-parser
channel {
junction {
channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };
}
#End Block parser generator app-parser
;
};
flags(final);
};
channel { flags(final); };
};
}; # channel
#End Block source generator system
;
internal();
syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));
};
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Balazs Scheidler
Sent: 26 de junho de 2022 06:37
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] Local sources seem not to be working
Hi,
I haven't seen anything like this. We are reading the journal files using libsystemd.
Try to remove the syslog-ng persist file and check if reading the journal restarts.
Also there's can be two ways of local messages getting to syslog-ng,
1) /dev/log forwarding
2) reading the journal files
The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.
To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.
I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.
On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>> wrote:
Hi
Any news regarding this issue?
Making a recap of the findings:
* Using a Debian 10 buster with first release with 3.36.1;
* After some time "system()" source logs are not getting written to the destinations;
* The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations;
* One the things I noticed is that the socket to the journal seems to vanish during the error situation:
It seems that somehow syslog-ng in unable to read from linux journal.
Have you ever experienced this problem?
Do know what can be wrong with the system?
root at machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root 15r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 16u REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root mem REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root 6r REG 0,19 8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal<mailto:/run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal>
Thanks in advance,
Alex
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Alexandre Santos
Sent: 19 de maio de 2022 09:25
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] Local sources seem not to be working
Hi Szilard,
There is no filter:
source syslog_ng_src {
internal();
};
destination d_localfile_syslog_ng {
program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
flags(syslog-protocol)
suppress(5)
disk-buffer(
mem-buf-size(2097152)
disk-buf-size(4194304)
reliable(yes)
dir("/tmp")
)
);
};
log {
source(syslog_ng_src);
destination(d_localfile_syslog_ng);
flags(flow-control);
};
Thanks and Regards,
Alex
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Szilard Parrag (sparrag)
Sent: 19 de maio de 2022 08:59
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Local sources seem not to be working
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Alex,
We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.
Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?
Szilard
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=a2TZ40wm6cSogr9U8Cohka%2FM%2FJS25mF2oAKN5UqcyZA%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SB1BSl3EuYVAtN3Phj9wMWSv5jwjN0zJ%2BAb0ArWNYkg%3D&reserved=0>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=JG%2FfZBGTh6BuFq76QXUhXLpnrkxOVRhod9Cy3sRRhzU%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SldWSDsKiv7WXDjZMsYrUYiAMryqgnKX6zMqwVMN9C8%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20221031/aa527fdc/attachment-0001.htm>
More information about the syslog-ng
mailing list