[syslog-ng] Console logging stops after some time.
Attila Szakacs (aszakacs)
Attila.Szakacs at oneidentity.com
Wed Feb 24 08:57:51 UTC 2021
Hi,
You are right, the 10 minutes wait cannot be configured.
Would such an option in the config help you? We could add time-reopen() support for the usertty destination driver.
Cheers,
Attila
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Alexandre Santos <alexandre.rosas.santos at gmail.com>
Sent: Tuesday, February 23, 2021 5:12 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Console logging stops after some time.
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi Attila,
Thank you for your reply. It made me look at the output in a different way.
I do not see any errors, but I did find these messages while running in verbose mode:
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140752] Posting message to user terminal; user='root', line='/dev/ttyS0'
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140868] Posting message to user terminal; user='root', line='/dev/pts/0'
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140915] Posting message to user terminal; user='root', line='/dev/pts/1'
Feb 23 15:24:25 localhost syslog-ng[13742]: [2021-02-23T15:24:25.140926] Writing to the user terminal has blocked for writing, disabling for 10 minutes; user='*'
So from the code in github I see that if the system call write is not successful due to EAGAIN, the write to terminals is disabled for 10 minutes.
And it seems there is no way to disable this, right?
Here are the stats:
root at localhost:~# syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
center;;received;a;processed;1179
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;dropped;0
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;processed;223
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;queued;0
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;suppressed;0
dst.program;d_localfile_apps#0;/opt/infinera/thanos/local/bin/write_with_rotation.sh /var/log/apps.log 10 10;a;written;223
src.internal;s_src#1;;a;processed;0
src.internal;s_src#1;;a;stamp;0
destination;d_localfile_apps;;a;processed;223
center;;queued;a;processed;446
global;scratch_buffers_count;;a;queued;25769803777
global;payload_reallocs;;a;processed;2556
global;sdata_updates;;a;processed;0
destination;d_console_log;;a;processed;223
global;scratch_buffers_bytes;;a;queued;256
src.journald;s_src#0;journal;a;processed;1179
src.journald;s_src#0;journal;a;stamp;1614095636
source;s_src;;a;processed;1179
global;msg_clones;;a;processed;0
Thanks & Regards,
Alex
On Tue, Feb 23, 2021 at 12:41 PM Attila Szakacs (aszakacs) <Attila.Szakacs at oneidentity.com<mailto:Attila.Szakacs at oneidentity.com>> wrote:
Hi Alex,
Can you show us a `syslog-ng-ctl stats` output?
Also, I can see that you have internal() source enabled, are there any errors in it?
Thanks!
Cheers,
Attila
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Alexandre Santos <alexandre.rosas.santos at gmail.com<mailto:alexandre.rosas.santos at gmail.com>>
Sent: Friday, February 19, 2021 2:15 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] Console logging stops after some time.
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi All,
I am using a Debian10 with syslog-ng 3.19.
I have a simple configuration formed by a log to a file and a log to the console "usertty(*)". Please check the attached file.
After some time the logging to the console stops, but the logging to the file keeps working.
I used strace to debug this. (strace -p <syslog-ng-pid> -yy -f -s 256 -e trace=write)
root at localhost:~# w root
12:50:41 up 3:16, 5 users, load average: 0.21, 0.35, 0.37
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root ttyS0 - 09:36 45:18 0.02s 0.02s -bash
root pts/1 10.0.2.2 09:40 1.00s 0.26s 0.00s w root
root pts/4 10.0.2.2 09:56 24:08 3.03s 2.84s strace -p 23429 -yy -f -s 256 -e trace=write
root pts/5 10.0.2.2 10:26 22:01 0.01s 0.01s tail -f /var/log/apps.log
root pts/0 10.0.2.2 12:31 19:25 21.77s 0.00s sleep 60
root at localhost:~# strace -p 23429 -yy -f -s 256 -e trace=write
[pid 24379] write(26</dev/ttyS0<char 4:64>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(26</dev/pts/1<char 136:1>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398
[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398
[pid 24379] write(26</dev/pts/5<char 136:5>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(26</dev/pts/0<char 136:0>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107
[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136
[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136
[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136
[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136
After some time I stop seeing writes to the console's, and only see writes to the apps.log file.
[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf INFO {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145
[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf INFO {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145
It seems that due to the high amount of messages the logging to the console is discarded.
Can you help me to solve this,
Thanks in advance,
Alex
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7Cdb4c325d289c4b8f0f3708d8d815e7fa%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637496935898454673%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EbyVtmhZyP8WHJsqbrhdrebcPKcbSfG7kflDxTEwMmU%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7Cdb4c325d289c4b8f0f3708d8d815e7fa%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637496935898459661%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZycQYCrnjDUpPgtpG0FyzULvJqfpKLTAccNwRDyZtTc%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7Cdb4c325d289c4b8f0f3708d8d815e7fa%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637496935898464654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l8sjb53Ac75zU9mnvXS0t0cb1ubCFoOjUnT55Bvc8Ew%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210224/317c3bc8/attachment-0001.html>
More information about the syslog-ng
mailing list