feature request?: monitor open logs for deletion/rename
Something which I think would be an awesome feature would be to monitor open file destinations for the file being renamed or deleted, and then reopen the file in such an event. The benefit of this is that when doing log rotation, you don't have to SIGHUP syslog-ng to make it re-open files. It would also make it so that all destination buffers don't have to be flushed and reopened, just the single file destination. The only downside is portability as not all OSs support the same way of doing this. On linux this can be easily done through inotify, and it looks like the BSD equivalent is kqueue (though I have pretty much no BSD experience). For platforms which there isn't a good method, we could instead fall back to a simple polling. -Patrick
Patrick Hemmer <syslogng@stormcloud9.net> writes:
Something which I think would be an awesome feature would be to monitor open file destinations for the file being renamed or deleted, and then reopen the file in such an event. The benefit of this is that when doing log rotation, you don't have to SIGHUP syslog-ng to make it re-open files. It would also make it so that all destination buffers don't have to be flushed and reopened, just the single file destination.
The only downside is portability as not all OSs support the same way of doing this. On linux this can be easily done through inotify, and it looks like the BSD equivalent is kqueue (though I have pretty much no BSD experience). For platforms which there isn't a good method, we could instead fall back to a simple polling.
I've been thinking about something similar recently, though, my desire started from a completely different angle: I'd love to have wildcard file sources, and possibly other stuff (like allow some macros in file sources, though that opens up a nasty can of worms). That needs some kind of monitoring too, and if sources have it, we can reuse the same thing for similar tasks on the destination side too. I planned to write an RFC (with a little bit more detail about how I imagine it would work, and what good things it'd bring us) about this in the next day or two, but the rabbit's out the hat now. On the flip side, I originally didn't think about how file monitoring could be used for destinations, but your mail enlightened me - thank you! -- |8]
hi, no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number. i was thinking about a separate operation to reopen files (e.g. syslog-ng-ctl reopen-files) to avoid having to reload the configuration, but this idea seems simpler to implement. ----- Original message -----
Patrick Hemmer <syslogng@stormcloud9.net> writes:
Something which I think would be an awesome feature would be to monitor open file destinations for the file being renamed or deleted, and then reopen the file in such an event. The benefit of this is that when doing log rotation, you don't have to SIGHUP syslog-ng to make it re-open files. It would also make it so that all destination buffers don't have to be flushed and reopened, just the single file destination.
The only downside is portability as not all OSs support the same way of doing this. On linux this can be easily done through inotify, and it looks like the BSD equivalent is kqueue (though I have pretty much no BSD experience). For platforms which there isn't a good method, we could instead fall back to a simple polling.
I've been thinking about something similar recently, though, my desire started from a completely different angle: I'd love to have wildcard file sources, and possibly other stuff (like allow some macros in file sources, though that opens up a nasty can of worms).
That needs some kind of monitoring too, and if sources have it, we can reuse the same thing for similar tasks on the destination side too.
I planned to write an RFC (with a little bit more detail about how I imagine it would work, and what good things it'd bring us) about this in the next day or two, but the rabbit's out the hat now. On the flip side, I originally didn't think about how file monitoring could be used for destinations, but your mail enlightened me - thank you!
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Balazs Scheidler <bazsi77@gmail.com> writes:
no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number.
But inotify has the benefit of not needing regular polls, and immediate notification when files change. I wouldn't want it for performance, because it wouldn't give a noticable boost in this case. I like inotify better in this case because of responsiveness, and effectiveness. For example, if my low-power system is happily idling, I'd rather not have syslog-ng regularly stat files and wake up the disk. However, if the kernel signals us that hey, this stuff here changed, that's much more efficient. Sadly, I don't have such a system where it would make any difference, but still! :P
i was thinking about a separate operation to reopen files (e.g. syslog-ng-ctl reopen-files) to avoid having to reload the configuration, but this idea seems simpler to implement.
syslog-ng-ctl reopen-files could be useful nevertheless, for different purposes. -- |8]
On Thu, Jul 12, 2012 at 11:40 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Balazs Scheidler <bazsi77@gmail.com> writes:
no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number.
But inotify has the benefit of not needing regular polls, and immediate notification when files change. I wouldn't want it for performance, because it wouldn't give a noticable boost in this case. I like inotify better in this case because of responsiveness, and effectiveness.
inotify works only on local filesystems - so you'll either end up adding some complexity (syslog-ng needs to check that a given destination file is on a local fs or on a remote one) or should use an existing framework having FAM-like functionality. glib has gio for such purpose, right?
Sandor Geller <Sandor.Geller@morganstanley.com> writes:
On Thu, Jul 12, 2012 at 11:40 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Balazs Scheidler <bazsi77@gmail.com> writes:
no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number.
But inotify has the benefit of not needing regular polls, and immediate notification when files change. I wouldn't want it for performance, because it wouldn't give a noticable boost in this case. I like inotify better in this case because of responsiveness, and effectiveness.
inotify works only on local filesystems - so you'll either end up adding some complexity (syslog-ng needs to check that a given destination file is on a local fs or on a remote one) or should use an existing framework having FAM-like functionality. glib has gio for such purpose, right?
There are solutions for it, yes, and care should be taken to only use inotify where it is appropriate - but where it is useful, it should be used over polling. However, I have not checked what glib has to offer, but I'm sure there's multiple solutions out there to aid us in this quest. -- |8]
Sent: Thu Jul 12 2012 06:12:11 GMT-0400 (EDT) From: Sandor Geller <Sandor.Geller@morganstanley.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] feature request?: monitor open logs for deletion/rename
On Thu, Jul 12, 2012 at 11:40 AM, Gergely Nagy<algernon@balabit.hu> wrote:
Balazs Scheidler<bazsi77@gmail.com> writes:
no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number. But inotify has the benefit of not needing regular polls, and immediate notification when files change. I wouldn't want it for performance, because it wouldn't give a noticable boost in this case. I like inotify better in this case because of responsiveness, and effectiveness. inotify works only on local filesystems - so you'll either end up adding some complexity (syslog-ng needs to check that a given destination file is on a local fs or on a remote one) or should use an existing framework having FAM-like functionality. glib has gio for such purpose, right? Actually inotify does work on network filesystems, but it just cant know when a remote host updates the file, only the local host. But nobody ever has multiple hosts writing to the same network file at the same time, right? Right? Just kidding, I know out there someone is going to be doing something evil like that.
-Patrick
----- Original message -----
Balazs Scheidler <bazsi77@gmail.com> writes:
no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number.
But inotify has the benefit of not needing regular polls, and immediate notification when files change. I wouldn't want it for performance, because it wouldn't give a noticable boost in this case. I like inotify better in this case because of responsiveness, and effectiveness.
For example, if my low-power system is happily idling, I'd rather not have syslog-ng regularly stat files and wake up the disk. However, if the kernel signals us that hey, this stuff here changed, that's much more efficient.
Sadly, I don't have such a system where it would make any difference, but still! :P
sorry I may have put that a bit easy to misunderstand. I meant that we can issue a stat before each write. No need to start separate polling. Syslog-ng has woken up in that case anyway.
i was thinking about a separate operation to reopen files (e.g. syslog-ng-ctl reopen-files) to avoid having to reload the configuration, but this idea seems simpler to implement.
syslog-ng-ctl reopen-files could be useful nevertheless, for different purposes.
Sent: Thu Jul 12 2012 07:07:44 GMT-0400 (EDT) From: Balazs Scheidler <bazsi77@gmail.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>, Gergely Nagy <algernon@balabit.hu> Subject: Re: [syslog-ng] feature request?: monitor open logs for deletion/rename
----- Original message -----
Balazs Scheidler <bazsi77@gmail.com <mailto:bazsi77@gmail.com>> writes:
no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number.
But inotify has the benefit of not needing regular polls, and immediate notification when files change. I wouldn't want it for performance, because it wouldn't give a noticable boost in this case. I like inotify better in this case because of responsiveness, and effectiveness.
For example, if my low-power system is happily idling, I'd rather not have syslog-ng regularly stat files and wake up the disk. However, if the kernel signals us that hey, this stuff here changed, that's much more efficient.
Sadly, I don't have such a system where it would make any difference, but still! :P
sorry I may have put that a bit easy to misunderstand. I meant that we can issue a stat before each write. No need to start separate polling. Syslog-ng has woken up in that case anyway.
I'd throw in support on this idea. The only downside I can see is if a log was rotated, the file handle wont be closed until the next write (so it'll still consume space on the filesystem), but syslog-ng will close the file after its been sitting idle for a while, so that might be good enough. -Patrick
On Thu, Jul 12, 2012 at 11:22:12AM +0200, Balazs Scheidler wrote:
no need to use inotify for this. merely stating the file regularly should indicate the new file with a changed inode number.
I have a 80%ish finished ivykis module (called iv_path_watch) that seems useful for this -- you hand it a path name, and it will call your callback if the path's contents may have changed. If inotify is available, it will use that, otherwise it will just periodically stat the path, which is basically the same idea as in iv_event_raw, which uses eventfds or pipes depending on which is available. The inotify case is particularly tricky to get 100% right: you have to look for not just writes to the file, but also for renames over the file, but also, you have to detect renames of any parent directories. And if your target is a symlink, you have to invoke yourself recursively. And then there are cases when inotify won't work at all, for example if the target is on a network filesystem. And then there's cases like chroot...
participants (5)
-
Balazs Scheidler
-
Gergely Nagy
-
Lennert Buytenhek
-
Patrick Hemmer
-
Sandor Geller