[syslog-ng] Local sources seem not to be working

Tue Mar 22 11:59:46 UTC 2022

Hi Gabor,

Thanks for the follow up and check my answers bellow in inline with my last email.

Some more details about the setup and another test it was done.
The system is running two syslog-ng instances, one in the default VRF and other in an Outer VRF.
syslog-ng -------------- uds socket ------------------> mgmt-syslog-ng -------- UDP ---------> [Log Server]
The syslog-ng in the default VRF is sending logs to the syslog-ng running in the outer VRF via Unix Domain Socket (destination d_mgmt_vrf_socket).
The mgmt-syslog-ng is running in the outer VRF and sending logs to the outside world.
Only the syslog-ng in the default VRF is reading sources internal and system.

We tested without having the remote logging (destination d_mgmt_vrf_socket) in the syslog-ng, and the problem did not appeared.

Hope this can give some enlightening about the problem.

Thanks & Regards,
Alex

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
Sent: 17 de março de 2022 20:09
To: Alexandre Santos <ASantos at infinera.com>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: Local sources seem not to be working

You are right, there is no flow-control for the log path where d_mgmt_vrf_socket destination is, I'm sorry.
Still, the internal log messages that the disk-buffer of d_mgmt_vrf_socket is filled are correct, but the source is not suspended.

I have some trouble understanding the problem, can you explain it please?
You're saying that the syslog() source in s_src is receiving message, while the internal() and system() doesn't?
[Alexandre Santos] Yes, I think that it is what is happening. Logs from syslog() source are being written to the /var/logs/..., while journald logs are not.
You've also stated that journald logs are working fine.
Does that mean that you can see new logs in journal, but not in syslog-ng?
[Alexandre Santos] Yes.

When the issue happens, can you check that internal() is working, e.g. by turning on and off the verbosity logging with "syslog-ng-ctl verbose --set on" and then "sbin/syslog-ng-ctl verbose --set off", please?
[Alexandre Santos] I saw no logs when I did this in error condition so I assume internal is not working as well.

This would generate an internal message with info level.
Also, can you check system() source as well with the "logger" command, e.g. "logger --rfc3164 test syslog-ng", please?
Could you give us a syslog-ng-ctl stats output too, please?
[Alexandre Santos] I have to this in the next test iteration.

Maybe I have found something, but I have to double-check: it looks like internal() source's messages are suppressed due to the destination d_mgmt_vrf_socket is unreachable:
<44>1 2022-03-11T11:52:45.313+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="4"] internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='Suppressing duplicate message; host=\'xmm4-1-1\', msg=\'Destination reliable queue full, dropping message; filename=\\'/tmp/syslog-ng-00016.rqf\\', queue_len=\\'6063\\', mem_buf_size=\\'2097152\\', disk_buf_size=\\'4194304\\', persist_name=\\'afsocket_dd_qfile(stream,localhost.afunix:/dev/uds_log)\\'\''
This means that there are no internal() logs until the destination is not reachable again.
Regards,
Gabor
________________________________
From: Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>>
Sent: Wednesday, March 16, 2022 16:53
To: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: RE: Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Gabor,

Thanks for the feedback.

But the flags(flow-control); is not set for the destination d_mgmt_vrf_socket. Only for the other destinations... d_localfile_<filename>.

That also does not explain the fact that log messages from:

syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no));

are still being written to the d_localfile_<filename>.

Any other idea?

Thanks in advance,

Alex

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>
Sent: 16 de março de 2022 15:09
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>; Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>>
Subject: Re: Local sources seem not to be working

Hi Alex!

I've checked the attached config and logs, and it looks like syslog-ng cannot send logs to the "/dev/uds_log" destination, and you have flow-control enabled in the config.

Once you fill the disk-buffer (which is a 4MiB sized reliable disk-buffer), flow-control kicks in and syslog-ng stops reading more messages from the sources that are connected to this destination.

example log:
Destination reliable queue full, dropping message; filename='/tmp/syslog-ng-00016.rqf', queue_len='6063', mem_buf_size='2097152', disk_buf_size='4194304', persist_name='afsocket_dd_qfile(stream,localhost.afunix:/dev/uds_log)'

At first, I would suggest to increase the disk-buffer size.

Regards,
Gabor

________________________________

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>>
Sent: Tuesday, March 15, 2022 16:04
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu> <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi,

I have syslog-ng 3.32.1 running in a Debian GNU/Linux 10 (buster) with the configuration in the attachement.

After sometime running, syslog-ng seems be unable to read from system() and internal() sources.

Log messages from syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no)); are seen in the output folders.

Also journald logs are working fine.

After a reload of configuration in which what changes is this line:

rewrite r_host { set("MACHINE-${HOST}", value("HOST")); };

logging is resumed.

Here is the time gap for logs:

<43>1 2022-03-11T11:55:23.802+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="767"] Last message 'Destination reliable' repeated 8933 times, suppressed by syslog-ng on xmm4-1-1

<46>1 2022-03-14T07:19:01.817+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="1"] Module loaded and initialized successfully; module='syslogformat'

Do you know why this is happening?

Thanks & Regards,

Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220322/d6322de9/attachment-0001.htm>