Patch for following logfiles
Sirs: Several months ago I submitted the following patch to syslog-ng, which makes the "file" source work even if the file is not present when syslog-ng starts up. (Consider the scenario where syslog-ng starts up before some other service which logs to a file, and that file has not yet been created yet.) This patch causes syslog-ng to treat the transition from "nonexistent" to "existent" the same as if the file were moved/renamed. We are using the patched version of syslog-ng in our project and find this modification helpful. It is a very non-intrusive change, and I cannot see any negative side-effects. I would like to know if there is any chance for this patch to be incorporated into the official distribution. Thanks for your consideration. Charles G Waldman High-Energy Physics University of Chicago cgw@hep.uchicago.edu Patch: (also available as http://repo.mwt2.org/viewvc/patches/syslog-ng-file-source.patch ) --- syslog-ng-2.0.4/src/affile.c 2007-04-19 14:37:16.000000000 -0500 +++ syslog-ng-2.0.4-cgw2/src/affile.c 2007-06-07 11:41:00.000000000 -0500 @@ -134,7 +134,7 @@ gint fd; msg_verbose("Follow-mode file source moved, tracking of the new file is started", - evt_tag_str("file", self->filename->str), + evt_tag_str("filename", self->filename->str), NULL); log_pipe_deinit(self->reader, NULL, NULL); @@ -170,14 +170,26 @@ { AFFileSourceDriver *self = (AFFileSourceDriver *) s; gint fd; + gboolean file_opened, open_deferred=FALSE; log_reader_options_init(&self->reader_options, cfg); - if (affile_sd_open_file(self, &fd)) + file_opened = affile_sd_open_file(self, &fd); + + if (!file_opened && self->reader_options.follow_freq > 0) + { + msg_info("Follow-mode file source not found, deferring open", + evt_tag_str("filename", self->filename->str), + NULL); + open_deferred = TRUE; + fd = -1; + } + + if (file_opened || open_deferred) { self->reader = log_reader_new(fd_read_new(fd, 0), LR_LOCAL | LR_NOMREAD, s, &self->reader_options); - if (persist) + if (persist && file_opened) { gchar *str; off_t cur_pos; --- syslog-ng-2.0.4/src/logreader.c 2007-04-20 15:11:07.000000000 -0500 +++ syslog-ng-2.0.4-cgw2/src/logreader.c 2007-06-07 11:26:12.000000000 -0500 @@ -100,29 +100,32 @@ struct stat st, followed_st; off_t pos; - pos = lseek(self->fd->fd, 0, SEEK_CUR); - if (pos == (off_t) -1) - { - msg_error("Error invoking seek on followed file", - evt_tag_errno("error", errno), - NULL); - return FALSE; - } - - if (fstat(self->fd->fd, &st) < 0) - { - msg_error("Error invoking fstat() on followed file", - evt_tag_errno("error", errno), - NULL); - return FALSE; - } + if (self->fd->fd >= 0) + { + pos = lseek(self->fd->fd, 0, SEEK_CUR); + if (pos == (off_t) -1) + { + msg_error("Error invoking seek on followed file", + evt_tag_errno("error", errno), + NULL); + return FALSE; + } + + if (fstat(self->fd->fd, &st) < 0) + { + msg_error("Error invoking fstat() on followed file", + evt_tag_errno("error", errno), + NULL); + return FALSE; + } - if (pos < st.st_size) - return TRUE; + if (pos < st.st_size) + return TRUE; + } if (self->reader->options->follow_filename && stat(self->reader->options->follow_filename, &followed_st) != -1) { - if (st.st_ino != followed_st.st_ino) + if (self->fd->fd < 0 || st.st_ino != followed_st.st_ino) { /* file was moved and we are at EOF, follow the new file */ log_pipe_notify(self->reader->control, &self->reader->super.super, NC_FILE_MOVED, self);
On Mon, 2007-12-03 at 17:15 -0600, Charles G Waldman wrote:
Sirs:
Several months ago I submitted the following patch to syslog-ng, which makes the "file" source work even if the file is not present when syslog-ng starts up. (Consider the scenario where syslog-ng starts up before some other service which logs to a file, and that file has not yet been created yet.) This patch causes syslog-ng to treat the transition from "nonexistent" to "existent" the same as if the file were moved/renamed.
We are using the patched version of syslog-ng in our project and find this modification helpful. It is a very non-intrusive change, and I cannot see any negative side-effects. I would like to know if there is any chance for this patch to be incorporated into the official distribution. Thanks for your consideration.
The patch is fine, could you please fax us a signed copy of the contributory license agreement on our webpage? Here's our contribution page that describes how to submit patches. http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/... Thanks. -- Bazsi
* Balazs Scheidler <bazsi@balabit.hu> [2007-12-04 09:30]:
[...] The patch is fine, could you please fax us a signed copy of the contributory license agreement on our webpage?
Just curious: you need that http://www.balabit.com/dl/CLA_patch.pdf in order to be able to include any contributions in your proprietary syslog-ng Premium Edition, which is entirely non-GPL, correct? -- Regards, Wolfram Schlich <wschlich@gentoo.org> Gentoo Linux * http://dev.gentoo.org/~wschlich/
On Tue, 2007-12-04 at 15:08 +0100, Wolfram Schlich wrote:
* Balazs Scheidler <bazsi@balabit.hu> [2007-12-04 09:30]:
[...] The patch is fine, could you please fax us a signed copy of the contributory license agreement on our webpage?
Just curious: you need that http://www.balabit.com/dl/CLA_patch.pdf in order to be able to include any contributions in your proprietary syslog-ng Premium Edition, which is entirely non-GPL, correct?
Correct. Just as we are backporting fixes from the Premium Edition, we'd also like to forward port patches from the Open Source edition. And we've started to work on the next major release of the Premium Edition, and we're going to backport complete features to the OSE. -- Bazsi
* Balazs Scheidler <bazsi@balabit.hu> [2007-12-04 17:09]:
On Tue, 2007-12-04 at 15:08 +0100, Wolfram Schlich wrote:
* Balazs Scheidler <bazsi@balabit.hu> [2007-12-04 09:30]:
[...] The patch is fine, could you please fax us a signed copy of the contributory license agreement on our webpage?
Just curious: you need that http://www.balabit.com/dl/CLA_patch.pdf in order to be able to include any contributions in your proprietary syslog-ng Premium Edition, which is entirely non-GPL, correct?
Correct. Just as we are backporting fixes from the Premium Edition, we'd also like to forward port patches from the Open Source edition.
And we've started to work on the next major release of the Premium Edition, and we're going to backport complete features to the OSE.
I see :) Thanks for the detailing! -- Regards, Wolfram Schlich <wschlich@gentoo.org> Gentoo Linux * http://dev.gentoo.org/~wschlich/
Dear Bazsi, while playing with the latest syslog-ng snapshot from Wed, 5 Dec 2007 10:57:01 +0000 http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=snapshot;h=c29352e3a3b61d... , I have received following error during my tests: "Internal error, duplicate configuration elements refer to the same persistent config; name='affile_sd_curpos(/aaa/01.txt)" for all non-existent log files. It is almost harmless, but should not be there. Here is the scenario: 1) The file /aaa/01.txt does not exists and is followed in my test configuration (same as I have sent some email before) 2) First start of syslog-ng ./syslog-ng -d -v -e -F -f /usr/local/etc/syslog-ng-tst.conf 3) Stopping of syslog-ng by Ctrl+C, everything looks fine, but, even the non-existent log files are written into the syslog-ng.persist ! 4) Second start, with same config, parameters and no other changes. 5) Second stop, and the error message mentioned above is generated. Also the error happens, in case you stop the syslog-ng, and the file previously stored in syslog-ng.persist does not exists anymore. I have also one more wish. Could you please make the syslog-ng.persist file: - configurable by command line option or in the config file? Sometimes it is better to move it to some other location, where syslog-ng can write and the recompilation is not comfortable here. (For example I prefer to run two syslog-ng processes on my system - one for log file reading as non-root and second as root as internal syslog replacement.) - And could you write persist file more frequently during the syslog-ng run, so the current file positions are not lost in case of machine crash or process hard termination? (I know, this is not the easy part due to possible performance impact.) I have also retested the patch "fixed flow-control problem related to source files" and it worked fine. Thank you. Best Regards, Pavel
On Fri, 2007-12-07 at 18:11 +0000, Behal, Pavel wrote:
Dear Bazsi,
while playing with the latest syslog-ng snapshot from Wed, 5 Dec 2007 10:57:01 +0000
http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=snapshot;h=c29352e3a3b61d...
, I have received following error during my tests:
"Internal error, duplicate configuration elements refer to the same persistent config; name='affile_sd_curpos(/aaa/01.txt)"
for all non-existent log files.
It is almost harmless, but should not be there. Here is the scenario:
1) The file /aaa/01.txt does not exists and is followed in my test configuration (same as I have sent some email before)
2) First start of syslog-ng ./syslog-ng -d -v -e -F -f /usr/local/etc/syslog-ng-tst.conf
3) Stopping of syslog-ng by Ctrl+C, everything looks fine, but, even the non-existent log files are written into the syslog-ng.persist !
4) Second start, with same config, parameters and no other changes.
5) Second stop, and the error message mentioned above is generated.
Also the error happens, in case you stop the syslog-ng, and the file previously stored in syslog-ng.persist does not exists anymore.
Thanks for reporting this, I've just pushed an update that fixes this issue.
I have also one more wish. Could you please make the syslog-ng.persist file:
- configurable by command line option or in the config file? Sometimes it is better to move it to some other location, where syslog-ng can write and the recompilation is not comfortable here. (For example I prefer to run two syslog-ng processes on my system - one for log file reading as non-root and second as root as internal syslog replacement.)
I've added a --persist-file command line option which does this.
- And could you write persist file more frequently during the syslog-ng run, so the current file positions are not lost in case of machine crash or process hard termination? (I know, this is not the easy part due to possible performance impact.)
This is not very easy, the whole concept of storing things persistently accross restarts relies on configuration initialization heavily. So this is impossible in the current design.
I have also retested the patch "fixed flow-control problem related to source files" and it worked fine. Thank you.
Thanks. -- Bazsi
On Mon, 10 Dec 2007, Balazs Scheidler wrote:
- And could you write persist file more frequently during the syslog-ng run, so the current file positions are not lost in case of machine crash or process hard termination? (I know, this is not the easy part due to possible performance impact.)
This is not very easy, the whole concept of storing things persistently accross restarts relies on configuration initialization heavily. So this is impossible in the current design.
I'm just thinking outloud here, but could the persist data be in the form of a syslog stream so that syslog-ng could treat it as a destination. Then all of the flush timeout settings apply, setings for where the destination is stored etc. Filters could be used to filter some type of internal message to this destination. This gives the user full control over performance tradeoffs regarding "up to date" and performance, and this would not require any redesign of the internals of syslog-ng. The only "gotcha" is that the user would be responsible for specifying a persistance source at startup that matches the persistence file specified in the configuration. Perhaps a reserved word for a destination that could be picked up during initialization. Just my $0.02 Evan Rempel
On Mon, 2007-12-03 at 17:15 -0600, Charles G Waldman wrote:
Sirs:
Several months ago I submitted the following patch to syslog-ng, which makes the "file" source work even if the file is not present when syslog-ng starts up. (Consider the scenario where syslog-ng starts up before some other service which logs to a file, and that file has not yet been created yet.) This patch causes syslog-ng to treat the transition from "nonexistent" to "existent" the same as if the file were moved/renamed.
We are using the patched version of syslog-ng in our project and find this modification helpful. It is a very non-intrusive change, and I cannot see any negative side-effects. I would like to know if there is any chance for this patch to be incorporated into the official distribution. Thanks for your consideration.
I've received your CLA, thanks. I'm integrating your patch right now.
Patch: (also available as http://repo.mwt2.org/viewvc/patches/syslog-ng-file-source.patch )
--- syslog-ng-2.0.4/src/affile.c 2007-04-19 14:37:16.000000000 -0500 +++ syslog-ng-2.0.4-cgw2/src/affile.c 2007-06-07 11:41:00.000000000 -0500 @@ -134,7 +134,7 @@ gint fd;
msg_verbose("Follow-mode file source moved, tracking of the new file is started", - evt_tag_str("file", self->filename->str), + evt_tag_str("filename", self->filename->str), NULL);
log_pipe_deinit(self->reader, NULL, NULL); @@ -170,14 +170,26 @@ { AFFileSourceDriver *self = (AFFileSourceDriver *) s; gint fd; + gboolean file_opened, open_deferred=FALSE;
log_reader_options_init(&self->reader_options, cfg);
- if (affile_sd_open_file(self, &fd)) + file_opened = affile_sd_open_file(self, &fd); + + if (!file_opened && self->reader_options.follow_freq > 0) + { + msg_info("Follow-mode file source not found, deferring open", + evt_tag_str("filename", self->filename->str), + NULL); + open_deferred = TRUE; + fd = -1; + } + + if (file_opened || open_deferred) { self->reader = log_reader_new(fd_read_new(fd, 0), LR_LOCAL | LR_NOMREAD, s, &self->reader_options);
- if (persist) + if (persist && file_opened) { gchar *str; off_t cur_pos; --- syslog-ng-2.0.4/src/logreader.c 2007-04-20 15:11:07.000000000 -0500 +++ syslog-ng-2.0.4-cgw2/src/logreader.c 2007-06-07 11:26:12.000000000 -0500 @@ -100,29 +100,32 @@ struct stat st, followed_st; off_t pos;
- pos = lseek(self->fd->fd, 0, SEEK_CUR); - if (pos == (off_t) -1) - { - msg_error("Error invoking seek on followed file", - evt_tag_errno("error", errno), - NULL); - return FALSE; - } - - if (fstat(self->fd->fd, &st) < 0) - { - msg_error("Error invoking fstat() on followed file", - evt_tag_errno("error", errno), - NULL); - return FALSE; - } + if (self->fd->fd >= 0) + { + pos = lseek(self->fd->fd, 0, SEEK_CUR); + if (pos == (off_t) -1) + { + msg_error("Error invoking seek on followed file", + evt_tag_errno("error", errno), + NULL); + return FALSE; + } + + if (fstat(self->fd->fd, &st) < 0) + { + msg_error("Error invoking fstat() on followed file", + evt_tag_errno("error", errno), + NULL); + return FALSE; + }
- if (pos < st.st_size) - return TRUE; + if (pos < st.st_size) + return TRUE; + }
if (self->reader->options->follow_filename && stat(self->reader->options->follow_filename, &followed_st) != -1) { - if (st.st_ino != followed_st.st_ino) + if (self->fd->fd < 0 || st.st_ino != followed_st.st_ino) { /* file was moved and we are at EOF, follow the new file */ log_pipe_notify(self->reader->control, &self->reader->super.super, NC_FILE_MOVED, self);
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Bazsi
participants (5)
-
Balazs Scheidler
-
Behal, Pavel
-
Charles G Waldman
-
Evan Rempel
-
Wolfram Schlich