We are running syslog-ng 3.16.1 on Centos 7.4.1708 on a central
logging host. We have a large (nearly 1,000) servers also running
the
same version of syslog-ng on the same Centos release. The servers
are
configured to log locally and also forward logs to the central
logging
host.
This morning we encountered a problem - syslog-ng was running on
the
logging host, but was not processing incoming logs or locally
generated ones. The servers forwarding to the central host did not
write anything to their local log files, a small but significant
portion of them had syslog crash, after which is was restarted by
systemd, but still no logs were written until syslog-ng was
forcibly
stopped on the central server and then restarted.
Connections to the central server weren't failing in the sense of
TCP
close or reset, but logs were accumulating on all the servers,
including the central one, in the cache file for buffering logs.
For our purposes, we need to have up-to-the moment logs available
on
the individual servers, so an admin going in to troubleshoot on a
server who only has console access still has recent logs to
consult if
needed.
Is there a way to tell syslog-ng to write local logs immediately
even
if it's currently buffering logs for sending to a non-responsive
remote server?