[syslog-ng] Get local log files written immediately even if remote log server is unresponsive

[Evan Rempel]

I noticed that between 3.9 and 3.14 this issue was introduced. Buffering to any destination seems to block all destinations of he message.

I am sure this did not work this way for 3.9 and earlier.

I didn't report this because I had not had the time to verify and test this behaviour.

Evan.
________________________________________
From: syslog-ng [syslog-ng-bounces at lists.balabit.hu] on behalf of Jim Segrave [jes at j-e-s.net]
Sent: Friday, July 5, 2019 6:58 AM
To: syslog-ng at lists.balabit.hu
Subject: [syslog-ng] Get local log files written immediately even if remote log server is unresponsive

We are running syslog-ng 3.16.1 on Centos 7.4.1708 on a central
logging host. We have a large (nearly 1,000) servers also running the
same version of syslog-ng on the same Centos release. The servers are
configured to log locally and also forward logs to the central logging
host.

This morning we encountered a problem - syslog-ng was running on the
logging host, but was not processing incoming logs or locally
generated ones. The servers forwarding to the central host did not
write anything to their local log files, a small but significant
portion of them had syslog crash, after which is was restarted by
systemd, but still no logs were written until syslog-ng was forcibly
stopped on the central server and then restarted.

Connections to the central server weren't failing in the sense of TCP
close or reset, but logs were accumulating on all the servers,
including the central one, in the cache file for buffering logs.

For our purposes, we need to have up-to-the moment logs available on
the individual servers, so an admin going in to troubleshoot on a
server who only has console access still has recent logs to consult if
needed.

Is there a way to tell syslog-ng to write local logs immediately even
if it's currently buffering logs for sending to a non-responsive
remote server?