[syslog-ng] file follow and new-file detection logic

[Evan Rempel]

Balazs Scheidler wrote:
> On Wed, 2008-03-12 at 08:33 -0700, Evan Rempel wrote:
>> syslog-ng 2.0.8
>>
>> I am just starting to use the "file" source to follow files and
>> would like to make a suggestion as to how this mechanism should work.
>>
>> Currently I understand that on startup, syslog-ng reads the persistent
>> data file to determine where is "left off" in a file and resumes from that location.
>> My concern is what it does if the file is no longer the same file.
>>
>> It seems that in the case where the file has changed, that none of the existing
>> data in the file is processed by syslog-ng, and only new data arriving in the file
>> is processed.
>>
>> I propose that the information saved in the persistent file should include
>> the offset within the file, and upon starting, if the offset does not match
>> a line break, then the file is no longer the same file and should be processed
>> from the beginning. Also, if the file is smaller than the offset, it is a new file
>> and should be processed from the beginning.
> 
> With the new serialize framework in 2.1 I can add the inode number of
> the file to the persistent information.
> 
> This way I can decide whether it's the same file (same inode) or not.
> 
> Does that sound ok for you?

Hmmm. Mostly good. I wouldn't do this, but if someone overwrote the file on startup,
and then started syslog-ng to read it, the first group of messages would still be
missed by syslog-ng. Perhaps /var/log/dmesg is an example. Another might be a redirected console.


I think that some logic to look at the size of the file, and if the persistent offset
makes sense given the data at that offset would be a more conclusive. Using the inode
may still be the most efficient if it has changed, but it is a startup cost, not a normal
running cost so efficiency may not be that important for this operation.

To be really sure, you could CRC check the last couple of lines that you have read from
that source and save that in your persistent data. Then on startup, read back two lines
and CRC check those to match your persistnet data. That way you can be very sure that the
file is or is not the same as the one your were using at the time syslog-ng shut down.


Evan Rempel