[syslog-ng] Behavior when followed files are renamed

[liste ütesi]

Evan Rempel wrote:
> Balazs Scheidler wrote:
>   
>> On Thu, 2008-12-11 at 18:01 -0500, Joe Shaw wrote:
>>     
>>> Hi,
>>>
>>> On Thu, Dec 11, 2008 at 5:37 PM, Balazs Scheidler <bazsi at balabit.hu> wrote:
>>>       
>>>> On Thu, 2008-12-11 at 15:23 -0500, Joe Shaw wrote:
>>>>         
>>>>> I couldn't find this anywhere in the online docs.  What's the behavior
>>>>> of syslog-ng OSE when a file which is being followed with
>>>>> follow_freq() is renamed?
>>>>>           
>>>> It will be read until EOF, and then if a new file happens to be there at
>>>> the same location, syslog-ng starts reading the new file. e.g. it
>>>> handles rotated logfiles just fine. Although I don't know if this is
>>>> your intention or not.
>>>>         
>>> Ah, ok, thanks for the info!  I think this is probably what I want,
>>> but I am worried about a race condition.  Consider the following
>>> scenario:
>>>
>>> 1. Some app is writing to a log file, and another program is
>>> monitoring when the logs should be rolled.
>>>
>>> 2. The log rolling program renames the file on disk, and the app
>>> continues to write to the renamed file because it still has a file
>>> descriptor open to it.
>>>
>>> 3. The log rolling program signals to the app to close and reopen its
>>> log file; the app is writing out to its fixed log file location again.
>>>
>>> Assuming that syslog-ng is monitoring the file, is there a possibility that:
>>>
>>> 1. The file is renamed
>>> 2. syslog-ng is fully caught up and hits the EOF
>>> 3. The new file isn't created yet, but syslog-ng is waiting for it to be created
>>> 4. The app continues to write some data out to the old, renamed file
>>> 5. The app is signaled and reopens the file; syslog-ng starts monitoring it
>>>
>>> I am making an assumption in #3 which might not be true -- that
>>> syslog-ng when it encounters EOF but the file isn't recreated that
>>> it'll abandon the renamed file.  If that's not true, then there's no
>>> race and I'm happy. :)
>>>       
>> It will not abandon it, but will check for the new file every
>> follow_freq() seconds.
>>     
>
> This would still seem to yield a race condition. Consider the following sequence;
>
> 1. Application is writing to log file named "A".
> 2. External log rotation renames "A" to "A.1"
> 3. External log rotation touches/creates file named "A" and sets appropriate
>     permissions.
> 4. Internal timer of syslog-ng is triggered by follow_freq() setting. Syslog-ng will
>     switch to the new file "A" because it exists, even though it was created only
>     milliseconds earlier.
> 5. log rotation signals the application to switch log files (reload or restart).
> 6. Application flushes log buffers to current file which is now A.1, but syslog-ng
>     is no longer reading this file.
> 7. Application closes current log file "A.1" and opens new log file "A".
>
> This sequence will result in the last buffer flush (step 6) from the application to
> be missed by syslog-ng.
>
> In my opinion, the only way to guarantee that you get all of the messages
> is to read from the old file until the first message appears in the new file.
> At that time, you can be assured that the old file has been abandoned by the
> application.
>
> Evan Rempel.
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>   
Just as an opinion,
I use syslog-ng 's internal variables for filenaming, so the rotaion is 
done internally by syslog-ng.

for example;
A-$YEAR-$MONTH-$DAY-$HOUR.log
works in my system

and then you can externally zip the old files