[syslog-ng] Local sources seem not to be working
Balazs Scheidler
bazsi77 at gmail.com
Tue Nov 1 05:36:02 UTC 2022
sorry, no I didn't.
I just looked at your configs, If I see correctly then in the error
situation the local logs are stalled, counters are not incremented. The
journald "processed" counter stays at 85558, while immediately after reload
it goes to 271377, which seems to indicate that for some reason we are not
getting updates on the journals.
I am reading the code to understand how we poll the journal and I hope I'll
have some input how to debug this.
Bazsi
On Mon, Oct 31, 2022 at 5:15 PM Alexandre Santos <ASantos at infinera.com>
wrote:
> Hi,
>
> Did you have a chance to analyze this?
>
> Thanks & Regards,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> * On Behalf Of *Alexandre
> Santos
> *Sent:* 16 de setembro de 2022 14:38
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> Hi Balazs,
>
>
>
> Do you have any other idea of how to debug this?
>
>
>
> From the last time I was able to reproduce the problem, I found the
> following:
>
>
>
> *New developments, on this.*
>
> *After another test this time with no remote destinations configured, the
> issue happened again:*
>
> *In attachment I am sending:*
>
> *- The stats in error condition: 42116.inerror.stats.txt,
> where ‘src.journald;s_src#0;journal;a;stamp;1651142246’*
>
> *- The stats in error condition after some time:
> 42116.inerror.15m.stats.txt where
> ‘src.journald;s_src#0;journal;a;stamp;1651142246’ *
>
> *- The stats after recovering the system with reload:
> 42116.after.stats.txt*
>
> *- The syslog-ng configuration:
> 42116.no.remote.dest.syslog-ng.conf*
>
>
>
> *It seems the ‘src.journald;s_src#0;journal;a;stamp;1651142246’did not
> changed. Does this means that last read timestamp from journal did not
> changed?*
>
> *Logs from the UDP source are still being written to the /var/log/… files.*
>
>
>
> Cheers,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> *On Behalf Of *Balazs
> Scheidler
> *Sent:* 1 de julho de 2022 21:02
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> This means that we are indeed polling the journal ourselves and not rely
> on journald to forward the logs to us.
>
>
>
> If the error occurs, try to look at the syslog-ng statistic counters
> (syslog-ng-ctl stats) to see if your output queue is full. This could cause
> back pressure and reading to stop.
>
>
>
> Also, enabling debug/trace messages of syslog-ng could be helpful. You can
> also do that via syslog-ng-ctl, no restart is needed. The trace output
> should be excluded from your normal log processing pipeline, as it can be
> quite overwhelming in volume.
>
>
>
> I don't really have any other ideas at the moment. Still on vacation. :)
>
>
>
> On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos at infinera.com> wrote:
>
> Hi Balazs,
>
>
>
> Thanks for the feedback!
>
>
>
> It is definitely using journald, as you can see below.
>
>
>
> When you say “Try to remove the syslog-ng persist file and check if
> reading the journal restarts.”, this is to do when the system in the error
> condition, right?
>
>
>
> Regards,
>
> Alex
>
>
>
> *source s_src {*
>
>
>
> *#Start Block source generator system*
>
> *channel {*
>
> * source {*
>
> *systemd-journal();*
>
>
>
> * }; # source*
>
> *channel {*
>
> * channel {*
>
> * parser {*
>
>
>
> *#Start Block parser generator app-parser*
>
>
>
> *channel {*
>
> * junction {*
>
>
>
> *channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };*
>
> *}*
>
> *#End Block parser generator app-parser*
>
> *;*
>
> * };*
>
> * flags(final);*
>
> * };*
>
> * channel { flags(final); };*
>
> *};*
>
> *}; # channel*
>
>
>
> *#End Block source generator system*
>
> *;*
>
> * internal();*
>
> * syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));*
>
> *};*
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> *On Behalf Of *Balazs
> Scheidler
> *Sent:* 26 de junho de 2022 06:37
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> Hi,
>
>
>
> I haven't seen anything like this. We are reading the journal files using
> libsystemd.
>
>
>
> Try to remove the syslog-ng persist file and check if reading the journal
> restarts.
>
>
>
> Also there's can be two ways of local messages getting to syslog-ng,
>
>
>
> 1) /dev/log forwarding
>
> 2) reading the journal files
>
>
>
> The first one is actively done by journald. Which one syslog-ng uses is
> automatically detected by our system() source.
>
>
>
> To see which one syslog-ng is trying to use, try to run it with
> --preprocess-into=some-file and check how system() source is expanded.
>
>
>
> I am unable to check the source code at the moment, so this is all from
> the top-of-my-head, but I hope this already helps to troubleshoot the issue.
>
>
>
> On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos at infinera.com> wrote:
>
> Hi
>
>
>
> Any news regarding this issue?
>
>
>
> Making a recap of the findings:
>
>
>
> - Using a Debian 10 buster with first release with 3.36.1;
>
>
> - After some time “system()” source logs are not getting written to
> the destinations;
> - The log messages from other sources, internal() and syslog(…)
> continue to work fine, being written to the destinations;
> - One the things I noticed is that the socket to the journal seems to
> vanish during the error situation:
>
>
>
> It seems that somehow syslog-ng in unable to read from linux journal.
>
> *Have you ever experienced this problem?*
>
> *Do know what can be wrong with the system?*
>
>
>
>
>
> root at machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>
> systemd-j 1723 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 16u REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 24u REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> syslog-ng 3201 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root 14r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root 15r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 5r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 6r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> root@ machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>
> systemd-j 1723 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 16u REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 24u REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 5r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 6r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
>
>
> Thanks in advance,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> *On Behalf Of *Alexandre
> Santos
> *Sent:* 19 de maio de 2022 09:25
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> Hi Szilard,
>
>
>
> There is no filter:
>
>
>
> source syslog_ng_src {
>
> internal();
>
> };
>
>
>
> destination d_localfile_syslog_ng {
>
> program("/opt/machine/local/bin/write_with_rotation.sh
> /var/log/syslog-ng-internal.log 10 10"
>
> flags(syslog-protocol)
>
> suppress(5)
>
> disk-buffer(
>
> mem-buf-size(2097152)
>
> disk-buf-size(4194304)
>
> reliable(yes)
>
> dir("/tmp")
>
> )
>
> );
>
> };
>
> log {
>
> source(syslog_ng_src);
>
> destination(d_localfile_syslog_ng);
>
> flags(flow-control);
>
> };
>
>
>
> Thanks and Regards,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> *On Behalf Of *Szilard
> Parrag (sparrag)
> *Sent:* 19 de maio de 2022 08:59
> *To:* syslog-ng at lists.balabit.hu
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Alex,
>
>
>
> We've checked it too and syslog-ng does not release the file descriptor of
> journald even with flow-control enabled.
>
>
>
> Also, your internal logs seem rather terse, maybe there is a filter which
> filters out the important parts. Could you please check it?
>
>
>
> Szilard
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0>
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=a2TZ40wm6cSogr9U8Cohka%2FM%2FJS25mF2oAKN5UqcyZA%3D&reserved=0>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SB1BSl3EuYVAtN3Phj9wMWSv5jwjN0zJ%2BAb0ArWNYkg%3D&reserved=0>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0>
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=JG%2FfZBGTh6BuFq76QXUhXLpnrkxOVRhod9Cy3sRRhzU%3D&reserved=0>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SldWSDsKiv7WXDjZMsYrUYiAMryqgnKX6zMqwVMN9C8%3D&reserved=0>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
--
Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20221101/d611a667/attachment-0001.htm>
More information about the syslog-ng
mailing list