On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng.
While looking through my pile of syslog-ng todo items I reread this message, and I missed something last time. Currently, syslog-ng falls back to 0.0.0.0 when resolving the DNS name fails, this in turn is interpreted by the kernel as 127.0.0.1 when used as a destination address. So that's the exact cause of the breakage. I have a half-baked patch that should fix this behaviour, however it does not sit too well with optional(yes) setting. (e.g. will fail even if optional is yes). Anyhow, here's the patch as it stands: http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=commit;h=6cd4fdaea7d77f1f... -- Bazsi