On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng.
I was already thinking about the idea of dropping internal messages generated while another internal message is being delivered to prevent such loops. I added this to my todo list. -- Bazsi