[syslog-ng] Local sources seem not to be working

Thu Mar 17 20:09:12 UTC 2022

You are right, there is no flow-control for the log path where d_mgmt_vrf_socket destination is, I'm sorry.
Still, the internal log messages that the disk-buffer of d_mgmt_vrf_socket is filled are correct, but the source is not suspended.

I have some trouble understanding the problem, can you explain it please?
You're saying that the syslog() source in s_src is receiving message, while the internal() and system() doesn't? You've also stated that journald logs are working fine.
Does that mean that you can see new logs in journal, but not in syslog-ng?

When the issue happens, can you check that internal() is working, e.g. by turning on and off the verbosity logging with "syslog-ng-ctl verbose --set on" and then "sbin/syslog-ng-ctl verbose --set off", please?
This would generate an internal message with info level.
Also, can you check system() source as well with the "logger" command, e.g. "logger --rfc3164 test syslog-ng", please?
Could you give us a syslog-ng-ctl stats output too, please?

Maybe I have found something, but I have to double-check: it looks like internal() source's messages are suppressed due to the destination d_mgmt_vrf_socket is unreachable:
<44>1 2022-03-11T11:52:45.313+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="4"] internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='Suppressing duplicate message; host=\'xmm4-1-1\', msg=\'Destination reliable queue full, dropping message; filename=\\'/tmp/syslog-ng-00016.rqf\\', queue_len=\\'6063\\', mem_buf_size=\\'2097152\\', disk_buf_size=\\'4194304\\', persist_name=\\'afsocket_dd_qfile(stream,localhost.afunix:/dev/uds_log)\\'\''

This means that there are no internal() logs until the destination is not reachable again.

Regards,
Gabor
________________________________
From: Alexandre Santos <ASantos at infinera.com>
Sent: Wednesday, March 16, 2022 16:53
To: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: RE: Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Gabor,

Thanks for the feedback.

But the flags(flow-control); is not set for the destination d_mgmt_vrf_socket. Only for the other destinations… d_localfile_<filename>.

That also does not explain the fact that log messages from:

syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no));

are still being written to the d_localfile_<filename>.

Any other idea?

Thanks in advance,

Alex

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
Sent: 16 de março de 2022 15:09
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>; Alexandre Santos <ASantos at infinera.com>
Subject: Re: Local sources seem not to be working

Hi Alex!

I've checked the attached config and logs, and it looks like syslog-ng cannot send logs to the "/dev/uds_log" destination, and you have flow-control enabled in the config.

Once you fill the disk-buffer (which is a 4MiB sized reliable disk-buffer), flow-control kicks in and syslog-ng stops reading more messages from the sources that are connected to this destination.

example log:
Destination reliable queue full, dropping message; filename='/tmp/syslog-ng-00016.rqf', queue_len='6063', mem_buf_size='2097152', disk_buf_size='4194304', persist_name='afsocket_dd_qfile(stream,localhost.afunix:/dev/uds_log)'

At first, I would suggest to increase the disk-buffer size.

Regards,
Gabor

________________________________

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Alexandre Santos <ASantos at infinera.com>
Sent: Tuesday, March 15, 2022 16:04
To: syslog-ng at lists.balabit.hu <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi,

I have syslog-ng 3.32.1 running in a Debian GNU/Linux 10 (buster) with the configuration in the attachement.

After sometime running, syslog-ng seems be unable to read from system() and internal() sources.

Log messages from syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no)); are seen in the output folders.

Also journald logs are working fine.

After a reload of configuration in which what changes is this line:

rewrite r_host { set("MACHINE-${HOST}", value("HOST")); };

logging is resumed.

Here is the time gap for logs:

<43>1 2022-03-11T11:55:23.802+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="767"] Last message 'Destination reliable' repeated 8933 times, suppressed by syslog-ng on xmm4-1-1

<46>1 2022-03-14T07:19:01.817+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="1"] Module loaded and initialized successfully; module='syslogformat'

Do you know why this is happening?

Thanks & Regards,

Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220317/79d662cc/attachment.htm>