Failure to source one file prevents entire syslog-ng logger from running
Hi, While I was away on holiday, another sysadmin changed the path of a 3rd party application logfile that I was I sourcing in syslog-ng (it cannot log directly to syslog). This caused the whole syslog-ng logger to basically fail to work/(re)start, giving an the error: Starting syslog-ng: Persistent configuration file is in invalid format, ignoring; Error opening file for reading; filename='/path/to/file.log', error='No such file or directory (2)' Error initializing source driver; source='s_file_blah' And auto-restarting the logger of course didn't work either since every time a restart is initiated the thing simply bailed out with this error. Shouldn't this error simply throw up a warning and continue logging the rest of what's happening on the system, rather than leaving a hole in the audit trail because it cannot lock in one source??? (Even better: start regardless, do what it can and then periodically check the source again, and every time it cannot get the source open, log a warning in syslog so that it can be caught by the monitoring infrastructure) Any suggestions for a workaround/option to prevent anything like this ever happening again in future would be welcome. -h -- Hari Sekhon Always open to interesting opportunities http://www.linkedin.com/in/harisekhon
On Tue, 2009-04-14 at 12:58 +0100, Hari Sekhon wrote:
Hi,
While I was away on holiday, another sysadmin changed the path of a 3rd party application logfile that I was I sourcing in syslog-ng (it cannot log directly to syslog). This caused the whole syslog-ng logger to basically fail to work/(re)start, giving an the error:
Starting syslog-ng: Persistent configuration file is in invalid format, ignoring; Error opening file for reading; filename='/path/to/file.log', error='No such file or directory (2)' Error initializing source driver; source='s_file_blah'
And auto-restarting the logger of course didn't work either since every time a restart is initiated the thing simply bailed out with this error.
Shouldn't this error simply throw up a warning and continue logging the rest of what's happening on the system, rather than leaving a hole in the audit trail because it cannot lock in one source??? (Even better: start regardless, do what it can and then periodically check the source again, and every time it cannot get the source open, log a warning in syslog so that it can be caught by the monitoring infrastructure)
Any suggestions for a workaround/option to prevent anything like this ever happening again in future would be welcome.
This is what syslog-ng 3.0 does by default. Every input file is checked every follow_freq() seconds and is reopened if it did not exist. -- Bazsi
Balazs Scheidler wrote:
On Tue, 2009-04-14 at 12:58 +0100, Hari Sekhon wrote:
Hi,
While I was away on holiday, another sysadmin changed the path of a 3rd party application logfile that I was I sourcing in syslog-ng (it cannot log directly to syslog). This caused the whole syslog-ng logger to basically fail to work/(re)start, giving an the error:
Starting syslog-ng: Persistent configuration file is in invalid format, ignoring; Error opening file for reading; filename='/path/to/file.log', error='No such file or directory (2)' Error initializing source driver; source='s_file_blah'
And auto-restarting the logger of course didn't work either since every time a restart is initiated the thing simply bailed out with this error.
Shouldn't this error simply throw up a warning and continue logging the rest of what's happening on the system, rather than leaving a hole in the audit trail because it cannot lock in one source??? (Even better: start regardless, do what it can and then periodically check the source again, and every time it cannot get the source open, log a warning in syslog so that it can be caught by the monitoring infrastructure)
Any suggestions for a workaround/option to prevent anything like this ever happening again in future would be welcome.
This is what syslog-ng 3.0 does by default. Every input file is checked every follow_freq() seconds and is reopened if it did not exist.
Ok, am on 2.x, will upgrade to 3.x. Does it log to syslog if the source is still unavailable every follow_freq() secs? This would be nice so that my monitoring infrastructure will alert me as to this problem rather than syslog-ng silently failing to pick up the source. Thanks -h -- Hari Sekhon Always open to interesting opportunities http://www.linkedin.com/in/harisekhon
On Wed, 2009-04-15 at 09:26 +0100, Hari Sekhon wrote:
Balazs Scheidler wrote:
On Tue, 2009-04-14 at 12:58 +0100, Hari Sekhon wrote:
Hi,
While I was away on holiday, another sysadmin changed the path of a 3rd party application logfile that I was I sourcing in syslog-ng (it cannot log directly to syslog). This caused the whole syslog-ng logger to basically fail to work/(re)start, giving an the error:
Starting syslog-ng: Persistent configuration file is in invalid format, ignoring; Error opening file for reading; filename='/path/to/file.log', error='No such file or directory (2)' Error initializing source driver; source='s_file_blah'
And auto-restarting the logger of course didn't work either since every time a restart is initiated the thing simply bailed out with this error.
Shouldn't this error simply throw up a warning and continue logging the rest of what's happening on the system, rather than leaving a hole in the audit trail because it cannot lock in one source??? (Even better: start regardless, do what it can and then periodically check the source again, and every time it cannot get the source open, log a warning in syslog so that it can be caught by the monitoring infrastructure)
Any suggestions for a workaround/option to prevent anything like this ever happening again in future would be welcome.
This is what syslog-ng 3.0 does by default. Every input file is checked every follow_freq() seconds and is reopened if it did not exist.
Ok, am on 2.x, will upgrade to 3.x.
Does it log to syslog if the source is still unavailable every follow_freq() secs? This would be nice so that my monitoring infrastructure will alert me as to this problem rather than syslog-ng silently failing to pick up the source.
Hi, It only has a log message in case the file is found, it does not log the interim attempts: msg_verbose("Follow-mode file source moved, tracking of the new file is started", evt_tag_str("filename", self->filename->str), NULL); You also need to enable verbose mode messages (-v) in order to see that. And this patch does exactly that: commit 478984b820e266c6a0d87f06cd7d22b84f6b7606 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Apr 22 13:03:26 2009 +0200 [logreader] print a log message if a polled file source does not exist diff --git a/src/logreader.c b/src/logreader.c index f9567ff..f1b3a7f 100644 --- a/src/logreader.c +++ b/src/logreader.c @@ -74,6 +74,7 @@ struct _LogReaderWatch LogReader *reader; GPollFD pollfd; LogProto *proto; + GTimeVal last_follow_freq_check; }; static gboolean @@ -178,6 +179,19 @@ log_reader_fd_check(GSource *source) log_pipe_notify(self->reader->control, &self->reader->super.super, NC_FILE_MOVED, self); } } + else if (self->reader->follow_filename) + { + GTimeVal now; + + g_source_get_current_time(source, &now); + if (g_time_val_diff(&now, &self->last_follow_freq_check) > self->reader->options->follow_freq * 1000) + { + msg_verbose("Follow mode file still does not exist", + evt_tag_str("filename", self->reader->follow_filename), + NULL); + self->last_follow_freq_check = now; + } + } return FALSE; } -- Bazsi
participants (2)
-
Balazs Scheidler
-
Hari Sekhon