[syslog-ng] Console logging stops after some time.

Attila Szakacs (aszakacs) Attila.Szakacs at oneidentity.com
Wed Feb 24 14:25:59 UTC 2021


I have opened a PR: https://github.com/syslog-ng/syslog-ng/pull/3585

Cheers,
Attila
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Alexandre Santos <alexandre.rosas.santos at gmail.com>
Sent: Wednesday, February 24, 2021 11:11 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Console logging stops after some time.

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Attila,

Yes, I think the config option time-reopen() for the usertty destination driver would make sense.

Thanks & Regards,
Alex

On Wed, Feb 24, 2021 at 8:57 AM Attila Szakacs (aszakacs) <Attila.Szakacs at oneidentity.com<mailto:Attila.Szakacs at oneidentity.com>> wrote:
Hi,

You are right, the 10 minutes wait cannot be configured.
Would such an option in the config help you? We could add time-reopen()​ support for the usertty destination driver.

Cheers,
Attila
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Alexandre Santos <alexandre.rosas.santos at gmail.com<mailto:alexandre.rosas.santos at gmail.com>>
Sent: Tuesday, February 23, 2021 5:12 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] Console logging stops after some time.

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Attila,

Thank you for your reply. It made me look at the output in a different way.
I do not see any errors, but I did find these messages while running in verbose mode:
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140752] Posting message to user terminal; user='root', line='/dev/ttyS0'
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140868] Posting message to user terminal; user='root', line='/dev/pts/0'
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140915] Posting message to user terminal; user='root', line='/dev/pts/1'
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140926] Writing to the user terminal has blocked for writing, disabling for 10 minutes; user='*'

So from the code in github I see that if the system call write is not successful due to EAGAIN, the write to terminals is disabled for 10 minutes.
And it seems there is no way to disable this, right?


Here are the stats:
root at localhost:~# syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
center;;received;a;processed;1179
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;dropped;0
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;processed;223
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;queued;0
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;suppressed;0
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;written;223
src.internal;s_src#1;;a;processed;0
src.internal;s_src#1;;a;stamp;0
destination;d_localfile_apps;;a;processed;223
center;;queued;a;processed;446
global;scratch_buffers_count;;a;queued;25769803777
global;payload_reallocs;;a;processed;2556
global;sdata_updates;;a;processed;0
destination;d_console_log;;a;processed;223
global;scratch_buffers_bytes;;a;queued;256
src.journald;s_src#0;journal;a;processed;1179
src.journald;s_src#0;journal;a;stamp;1614095636
source;s_src;;a;processed;1179
global;msg_clones;;a;processed;0

Thanks & Regards,
Alex


On Tue, Feb 23, 2021 at 12:41 PM Attila Szakacs (aszakacs) <Attila.Szakacs at oneidentity.com<mailto:Attila.Szakacs at oneidentity.com>> wrote:
Hi Alex,

Can you show us a `syslog-ng-ctl stats` output?
Also, I can see that you have internal() source enabled, are there any errors in it?

Thanks!

Cheers,
Attila
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Alexandre Santos <alexandre.rosas.santos at gmail.com<mailto:alexandre.rosas.santos at gmail.com>>
Sent: Friday, February 19, 2021 2:15 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] Console logging stops after some time.

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi All,

I am using a Debian10 with syslog-ng 3.19.

I have a simple configuration formed by a log to a file and a log to the console "usertty(*)". Please check the attached file.

After some time the logging to the console stops, but the logging to the file keeps working.

I used strace to debug this. (strace -p <syslog-ng-pid> -yy -f -s 256 -e trace=write)
root at localhost:~# w root
 12:50:41 up  3:16,  5 users,  load average: 0.21, 0.35, 0.37
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     ttyS0    -                09:36   45:18   0.02s  0.02s -bash
root     pts/1    10.0.2.2         09:40    1.00s  0.26s  0.00s w root
root     pts/4    10.0.2.2         09:56   24:08   3.03s  2.84s strace -p 23429 -yy -f -s 256 -e trace=write
root     pts/5    10.0.2.2         10:26   22:01   0.01s  0.01s tail -f /var/log/apps.log
root     pts/0    10.0.2.2         12:31   19:25  21.77s  0.00s sleep 60

root at localhost:~# strace -p 23429 -yy -f -s 256 -e trace=write

[pid 24379] write(26</dev/ttyS0<char 4:64>>, "2021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(26</dev/pts/1<char 136:1>>, "2021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398
[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398
[pid 24379] write(26</dev/pts/5<char 136:5>>, "2021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(26</dev/pts/0<char 136:0>>, "2021 Feb 19 12:31:16 localhost 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136
[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136
[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136
[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm             INFO   {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136

After some time I stop seeing writes to the console's, and only see writes to the apps.log file.

[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf             INFO   {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145
[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf             INFO   {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145


It seems that due to the high amount of messages the logging to the console is discarded.
Can you help me to solve this,
Thanks in advance,
Alex



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7C1819aa1e73f24983b6c108d8d8ac8578%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637497582774106127%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=t%2BtG%2FkHiHt%2F4t1hOtfAivBIPwIrPLiOMoHdUy8csnjM%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7C1819aa1e73f24983b6c108d8d8ac8578%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637497582774116111%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SeuaRXNktZFBcuBz8czNOK4SBd9QR3TGGxSj7FgRLKk%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7C1819aa1e73f24983b6c108d8d8ac8578%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637497582774121099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Fd9uzJLIXZHfDCosvmScKdAAGTQVMDL9Ll5w4CIZ%2BgM%3D&reserved=0>

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7C1819aa1e73f24983b6c108d8d8ac8578%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637497582774126093%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=OONFcz9RGwd9%2F%2BaofBSMjiqWFuScyEo1qENk%2FnVR0aQ%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7C1819aa1e73f24983b6c108d8d8ac8578%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637497582774131081%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vK%2F5CRI3sB7gmPuXrHrD%2FXBtahYsPL2TBKjiXPUu8Kg%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7C1819aa1e73f24983b6c108d8d8ac8578%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637497582774136072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MsCaLQc7sqwe0EDGKCCkuLs2U7bPqVKtZt6%2FGgfNMzc%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210224/7b11569c/attachment-0001.html>


More information about the syslog-ng mailing list