[syslog-ng] Local sources seem not to be working

Thu Apr 21 13:22:31 UTC 2022

Hi Gabor,

The problem was reproduced again.
I was able to get the stats in the error situation: 54146.stats.error.txt

I also took the stats after reloading the configuration (which fixes the problem): 54146.stats.after.txt

Regarding your question: Just a random question: is /tmp a tmpfs filesystem?
Yes it is.

Let me know what you found out.

Thanks and regards,
Alex

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
Sent: 31 de março de 2022 11:14
To: Alexandre Santos <ASantos at infinera.com>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: Local sources seem not to be working

Thanks for the config! I'll continue experimenting on my ideas

You could either configure syslog-ng-ctl stats to talk to a given syslog-ng instance with the --control option pointing to the control-socket e.g. as above /var/lib/syslog-ng/mgmt-syslog-ng.ctl,
OR use the syslog-ng-ctl instance under the 2nd syslog-ng installation path.

Just a random question: is /tmp a tmpfs filesystem?

Gabor
________________________________
From: Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>>
Sent: Wednesday, March 30, 2022 17:07
To: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: RE: Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Gabor,

Thank you for you feedback.

Can you share the config, when the issue cannot be seen?

I am sending the configuration in attachment.

I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.

The issue is hard to reproduce, the next time the error is seen, I try to run it.

But how can I run syslog-ng-ctl stats for the 2nd syslog-ng instance?

root at machine:/~# ps -ewfH | grep syslog-ng

root      2582     1  0 09:06 ?        00:00:34   /usr/sbin/syslog-ng -F --caps cap_net_bind_service,cap_net_broadcast,cap_net_raw,cap_dac_read_search,cap_chown,cap_fowner=p cap_dac_override,cap_syslog=ep

root      4018     1  0 09:07 ?        00:00:00   /usr/sbin/syslog-ng -F --cfgfile=/etc/syslog-ng/mgmt-syslog-ng.conf --pidfile=/var/lib/syslog-ng/mgmt-syslog-ng.pid --persist-file=/var/lib/syslog-ng/mgmt-syslog-ng.persist --control=/var/lib/syslog-ng/mgmt-syslog-ng.ctl

syslog-ng-ctl, seems to only show stats for job 2582.

Regards, Alex

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>
Sent: 29 de março de 2022 12:48
To: Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: Local sources seem not to be working

Hi Alex,

Using regular disk-buffer vs. using reliable disk-buffer shouldn't cause symptoms like that. It sounds like reliable(yes) would turn on a flow-control-like behaviour, which it doesn't.

(And as you said it only affects local sources).
The main difference between the two kinds of disk-buffers is, that while reliable disk-buffer write every message to the disk-buffer, a normal disk-buffer has memory-only buffers for performance reasons (and flow-control reasons too).

You can still lose logs with a reliable disk-buffer if no flow-control is used: when the disk-buffer has reached it's maximum size and new messages keep arriving, then syslog-ng drops those messages.

We have more detailed documentation about disk-buffers in the admin guide, where you can see the structure of disk-buffers:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.36%2Fadministration-guide%2F61%23TOPIC-1768724&data=04%7C01%7CASantos%40infinera.com%7Ca2a160c601fd486c362b08da12ff3fae%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637843184746371785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SMSL5ZoOMh1Kmw%2B35UUSFGuyNOJVB6NQQstm37ANkoI%3D&reserved=0>

Can you share the config, when the issue cannot be seen?

I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.

Regards,

Gabor

________________________________

From: Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>>
Sent: Monday, March 28, 2022 13:45
To: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: RE: Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Gabor,

"This is strange: the d_localfile destinations (as well as the vrf-socket destination "d_mgmt_vrf_socket") receive messages from the syslog() source, but not from the internal() or system() sources?"

Yes.

And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?

Yes.

I could not test the 2 last suggestions that you made.

We did however another test, which was to remove the reliable option from d_mgmt_vrf_socket, and it seems the problem is not seen again.

Besides from what it is written in the manual, in other which cases/conditions can syslog-ng loose logs?

reliable()

Type:

yes|no

Default:

no

Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.

Thanks in advance,

Alex

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>
Sent: 25 de março de 2022 14:44
To: Alexandre Santos <ASantos at infinera.com<mailto:ASantos at infinera.com>>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Alex,

Sorry I haven't answered yet. I'll have a few ideas I would like to try out next week.

This is strange: the d_localfile destinations (as well as the vrf-socket destination "d_mgmt_vrf_socket") receive messages from the syslog() source, but not from the internal() or system() sources?

And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?

If it would be soft flow-control, then the syslog() source would be suspended too.

Just a tip: would you switch out the unix-dgram() destination to syslog() destination, please? Maybe that's not possible with the VRF in-place...

In the stats output, do you see an increased number of dropped messages?

I would still suggest increasing the 4MB disk-buffer. You should make an estimation of how long could the mgmt syslog-ng be down (i.e not receiving from the unix-dgram), what is the average incoming EPS and an average message size, that could give a hint about the required disk-buffer size.

Regards,

Gabor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220421/603dfe60/attachment-0001.htm>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 45146.stats.error.txt
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220421/603dfe60/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 45146.stats.after.txt
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220421/603dfe60/attachment-0003.txt>