[syslog-ng] Local sources seem not to be working
Balazs Scheidler
bazsi77 at gmail.com
Fri Jul 1 20:02:09 UTC 2022
This means that we are indeed polling the journal ourselves and not rely on
journald to forward the logs to us.
If the error occurs, try to look at the syslog-ng statistic counters
(syslog-ng-ctl stats) to see if your output queue is full. This could cause
back pressure and reading to stop.
Also, enabling debug/trace messages of syslog-ng could be helpful. You can
also do that via syslog-ng-ctl, no restart is needed. The trace output
should be excluded from your normal log processing pipeline, as it can be
quite overwhelming in volume.
I don't really have any other ideas at the moment. Still on vacation. :)
On Thu, Jun 30, 2022, 19:43 Alexandre Santos <ASantos at infinera.com> wrote:
> Hi Balazs,
>
>
>
> Thanks for the feedback!
>
>
>
> It is definitely using journald, as you can see below.
>
>
>
> When you say “Try to remove the syslog-ng persist file and check if
> reading the journal restarts.”, this is to do when the system in the error
> condition, right?
>
>
>
> Regards,
>
> Alex
>
>
>
> *source s_src {*
>
>
>
> *#Start Block source generator system*
>
> *channel {*
>
> * source {*
>
> *systemd-journal();*
>
>
>
> * }; # source*
>
> *channel {*
>
> * channel {*
>
> * parser {*
>
>
>
> *#Start Block parser generator app-parser*
>
>
>
> *channel {*
>
> * junction {*
>
>
>
> *channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };*
>
> *}*
>
> *#End Block parser generator app-parser*
>
> *;*
>
> * };*
>
> * flags(final);*
>
> * };*
>
> * channel { flags(final); };*
>
> *};*
>
> *}; # channel*
>
>
>
> *#End Block source generator system*
>
> *;*
>
> * internal();*
>
> * syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));*
>
> *};*
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> * On Behalf Of *Balazs
> Scheidler
> *Sent:* 26 de junho de 2022 06:37
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> Hi,
>
>
>
> I haven't seen anything like this. We are reading the journal files using
> libsystemd.
>
>
>
> Try to remove the syslog-ng persist file and check if reading the journal
> restarts.
>
>
>
> Also there's can be two ways of local messages getting to syslog-ng,
>
>
>
> 1) /dev/log forwarding
>
> 2) reading the journal files
>
>
>
> The first one is actively done by journald. Which one syslog-ng uses is
> automatically detected by our system() source.
>
>
>
> To see which one syslog-ng is trying to use, try to run it with
> --preprocess-into=some-file and check how system() source is expanded.
>
>
>
> I am unable to check the source code at the moment, so this is all from
> the top-of-my-head, but I hope this already helps to troubleshoot the issue.
>
>
>
> On Fri, Jun 24, 2022, 18:21 Alexandre Santos <ASantos at infinera.com> wrote:
>
> Hi
>
>
>
> Any news regarding this issue?
>
>
>
> Making a recap of the findings:
>
>
>
> - Using a Debian 10 buster with first release with 3.36.1;
>
>
> - After some time “system()” source logs are not getting written to
> the destinations;
> - The log messages from other sources, internal() and syslog(…)
> continue to work fine, being written to the destinations;
> - One the things I noticed is that the socket to the journal seems to
> vanish during the error situation:
>
>
>
> It seems that somehow syslog-ng in unable to read from linux journal.
>
> *Have you ever experienced this problem?*
>
> *Do know what can be wrong with the system?*
>
>
>
>
>
> root at machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>
> systemd-j 1723 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 16u REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 24u REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> syslog-ng 3201 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root 14r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root 15r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 5r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 6r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> root@ machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>
> systemd-j 1723 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 16u REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root 24u REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root mem REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root mem REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 5r REG 0,19 8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root 6r REG 0,19 8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system at 3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
>
>
> Thanks in advance,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> *On Behalf Of *Alexandre
> Santos
> *Sent:* 19 de maio de 2022 09:25
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> Hi Szilard,
>
>
>
> There is no filter:
>
>
>
> source syslog_ng_src {
>
> internal();
>
> };
>
>
>
> destination d_localfile_syslog_ng {
>
> program("/opt/machine/local/bin/write_with_rotation.sh
> /var/log/syslog-ng-internal.log 10 10"
>
> flags(syslog-protocol)
>
> suppress(5)
>
> disk-buffer(
>
> mem-buf-size(2097152)
>
> disk-buf-size(4194304)
>
> reliable(yes)
>
> dir("/tmp")
>
> )
>
> );
>
> };
>
> log {
>
> source(syslog_ng_src);
>
> destination(d_localfile_syslog_ng);
>
> flags(flow-control);
>
> };
>
>
>
> Thanks and Regards,
>
> Alex
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> *On Behalf Of *Szilard
> Parrag (sparrag)
> *Sent:* 19 de maio de 2022 08:59
> *To:* syslog-ng at lists.balabit.hu
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Alex,
>
>
>
> We've checked it too and syslog-ng does not release the file descriptor of
> journald even with flow-control enabled.
>
>
>
> Also, your internal logs seem rather terse, maybe there is a filter which
> filters out the important parts. Could you please check it?
>
>
>
> Szilard
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=WNrM7LVg0OQZ4H%2FdhbZd4mCQ%2F6dAUsu9gJlgEa2aoCk%3D&reserved=0>
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=O5s9C1DJkKe%2F%2FJRWzDpRO4tpU1I31b%2F%2BG1SBXmG26Ps%3D&reserved=0>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C103d70dac2394363b6f808da5735e22c%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637918186202981595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=CiurkG9FrrVfxETCA8L5nVWoHofbzNrwn%2BP%2BpfAHX1k%3D&reserved=0>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220701/3e39ee69/attachment-0001.htm>
More information about the syslog-ng
mailing list