[syslog-ng] Local sources seem not to be working

[Gabor Nagy (gnagy)]

Hi Alex,

Using regular disk-buffer vs. using reliable disk-buffer shouldn't cause symptoms like that. It sounds like reliable(yes) would turn on a flow-control-like behaviour, which it doesn't.
(And as you said it only affects local sources).
The main difference between the two kinds of disk-buffers is, that while reliable disk-buffer write every message to the disk-buffer, a normal disk-buffer has memory-only buffers for performance reasons (and flow-control reasons too).
You can still lose logs with a reliable disk-buffer if no flow-control is used: when the disk-buffer has reached it's maximum size and new messages keep arriving, then syslog-ng drops those messages.

We have more detailed documentation about disk-buffers in the admin guide, where you can see the structure of disk-buffers:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724

Can you share the config, when the issue cannot be seen?
I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.

Regards,
Gabor
________________________________
From: Alexandre Santos <ASantos at infinera.com>
Sent: Monday, March 28, 2022 13:45
To: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: RE: Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


Hi Gabor,



“This is strange: the d_localfile destinations (as well as the vrf-socket destination "d_mgmt_vrf_socket") receive messages from the syslog() source, but not from the internal() or system() sources?”

Yes.



And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?

Yes.



I could not test the 2 last suggestions that you made.



We did however another test, which was to remove the reliable option from d_mgmt_vrf_socket, and it seems the problem is not seen again.



Besides from what it is written in the manual, in other which cases/conditions can syslog-ng loose logs?



reliable()

Type:

yes|no

Default:

no

Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.



Thanks in advance,

Alex



From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
Sent: 25 de março de 2022 14:44
To: Alexandre Santos <ASantos at infinera.com>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: Local sources seem not to be working



CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.



Hi Alex,

Sorry I haven't answered yet. I'll have a few ideas I would like to try out next week.

This is strange: the d_localfile destinations (as well as the vrf-socket destination "d_mgmt_vrf_socket") receive messages from the syslog() source, but not from the internal() or system() sources?

And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?

If it would be soft flow-control, then the syslog() source would be suspended too.

Just a tip: would you switch out the unix-dgram() destination to syslog() destination, please? Maybe that's not possible with the VRF in-place...

In the stats output, do you see an increased number of dropped messages?



I would still suggest increasing the 4MB disk-buffer. You should make an estimation of how long could the mgmt syslog-ng be down (i.e not receiving from the unix-dgram), what is the average incoming EPS and an average message size, that could give a hint about the required disk-buffer size.

Regards,

Gabor

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220329/bc111bb0/attachment.htm>