[syslog-ng] Failure to source one file prevents entire syslog-ng logger from running

[Balazs Scheidler]

On Wed, 2009-04-15 at 09:26 +0100, Hari Sekhon wrote:
> Balazs Scheidler wrote:
> > On Tue, 2009-04-14 at 12:58 +0100, Hari Sekhon wrote:
> >   
> >> Hi,
> >>
> >>    While I was away on holiday, another sysadmin changed the path of a 
> >> 3rd party application logfile that I was I sourcing in syslog-ng (it 
> >> cannot log directly to syslog). This caused the whole syslog-ng logger 
> >> to basically fail to work/(re)start, giving an the error:
> >>
> >> Starting syslog-ng: Persistent configuration file is in invalid format, 
> >> ignoring;
> >> Error opening file for reading; filename='/path/to/file.log', error='No 
> >> such file or directory (2)'
> >> Error initializing source driver; source='s_file_blah'
> >>
> >> And auto-restarting the logger of course didn't work either since every 
> >> time a restart is initiated the thing simply bailed out with this error.
> >>
> >> Shouldn't this error simply throw up a warning and continue logging the 
> >> rest of what's happening on the system, rather than leaving a hole in 
> >> the audit trail because it cannot lock in one source???
> >> (Even better: start regardless, do what it can and then periodically 
> >> check the source again, and every time it cannot get the source open, 
> >> log a warning in syslog so that it can be caught by the monitoring 
> >> infrastructure)
> >>
> >> Any suggestions for a workaround/option to prevent anything like this 
> >> ever happening again in future would be welcome.
> >>     
> >
> > This is what syslog-ng 3.0 does by default. Every input file is checked
> > every follow_freq() seconds and is reopened if it did not exist.
> >   
> Ok, am on 2.x, will upgrade to 3.x.
> 
> Does it log to syslog if the source is still unavailable every 
> follow_freq() secs? This would be nice so that my monitoring 
> infrastructure will alert me as to this problem rather than syslog-ng 
> silently failing to pick up the source.

Hi,

It only has a log message in case the file is found, it does not log the
interim attempts:

        msg_verbose("Follow-mode file source moved, tracking of the new file is started",
                    evt_tag_str("filename", self->filename->str),
                    NULL);

You also need to enable verbose mode messages (-v) in order to see that. And this patch 
does exactly that:

commit 478984b820e266c6a0d87f06cd7d22b84f6b7606
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Wed Apr 22 13:03:26 2009 +0200

    [logreader] print a log message if a polled file source does not exist

diff --git a/src/logreader.c b/src/logreader.c
index f9567ff..f1b3a7f 100644
--- a/src/logreader.c
+++ b/src/logreader.c
@@ -74,6 +74,7 @@ struct _LogReaderWatch
   LogReader *reader;
   GPollFD pollfd;
   LogProto *proto;
+  GTimeVal last_follow_freq_check;
 };
 
 static gboolean
@@ -178,6 +179,19 @@ log_reader_fd_check(GSource *source)
               log_pipe_notify(self->reader->control, &self->reader->super.super, NC_FILE_MOVED, self);
             }
         }
+      else if (self->reader->follow_filename)
+        {
+          GTimeVal now;
+
+          g_source_get_current_time(source, &now);
+          if (g_time_val_diff(&now, &self->last_follow_freq_check) > self->reader->options->follow_freq * 1000)
+            {
+              msg_verbose("Follow mode file still does not exist",
+                          evt_tag_str("filename", self->reader->follow_filename),
+                          NULL);
+              self->last_follow_freq_check = now;
+            }
+        }
       return FALSE;
     }
     



-- 
Bazsi