Followed files that grow while syslog-ng is reloaded
Hi list, I'm using syslog-ng (3.2.4) to follow some nginx log files using these directives: source s_nginx_logs { file("/path/to/nginx/access.log" flags(no-parse)); } When syslog-ng is reloaded, either for its own log files rotation or because a configuration update (which happens regularly when a new site is added), it does not see new data being written in followed files while it is reloading itself, and then complains about file size having changed and retransmit the whole file to my central log file collector. Here is the warning message: syslog-ng[24608]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/path/to/nginx/access.log) stored_inode='139419', cur_file_inode='139419', stored_size='985114', cur_file_size='997815', raw_stream_pos='1995630' I don't see how to handle that correctly. Any idea ?
Hi, I don't see that the fact the file is growing while syslog-ng is being reloaded would cause something like this. it is designed to handle these cases properly. I even tried to reproduce the issue with my 3.4 tree and couldn't. There was one related change in 3.2.5 though: Author: Balazs Scheidler <bazsi@balabit.hu> 2011-05-29 19:47:19 Committer: Balazs Scheidler <bazsi@balabit.hu> 2011-10-08 12:33:16 Parent: e231e957d13af412f38db817e2621a4ece8ebe8a (LogProto: apply_state shouldn't allow file offsets over the end-of-file) Child: 32650278fb35fb6f82522df54014c8d57f1705aa ([persist]: only check the buffer size if no encoding is specified) Branches: master, remotes/origin/master Follows: v3.2.4 Precedes: v3.2.5 LogProtoBufferedServer: fixed error handling in case the persist-state cannot be restored In case an error happens while restoring the persist data, be sure to reset various state variables to indicate that the Protocol is to be restarted. Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> But that would only trigger the error message if for some reason syslog-ng couldn't restore the state earlier. On Sat, Jun 30, 2012 at 5:16 PM, Francois Durand <fdur559@gmail.com> wrote:
Hi list,
I'm using syslog-ng (3.2.4) to follow some nginx log files using these directives:
source s_nginx_logs { file("/path/to/nginx/access.log" flags(no-parse)); }
When syslog-ng is reloaded, either for its own log files rotation or because a configuration update (which happens regularly when a new site is added), it does not see new data being written in followed files while it is reloading itself, and then complains about file size having changed and retransmit the whole file to my central log file collector.
Here is the warning message: syslog-ng[24608]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/path/to/nginx/access.log) stored_inode='139419', cur_file_inode='139419', stored_size='985114', cur_file_size='997815', raw_stream_pos='1995630'
I don't see how to handle that correctly.
Any idea ?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Hi, On Wed, Jul 4, 2012 at 9:47 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
I don't see that the fact the file is growing while syslog-ng is being reloaded would cause something like this. it is designed to handle these cases properly.
Log rotation happens at 6:25. This error line happens exactly once a day, at 6:25:02 (it is always the first line of the newly created syslog file). The error message is quite clear about what syslog-ng is doing (and indeed that's what is happening) and why it is doing so (the filesizes reported are plausible. But I only assumed syslog-ng was not lying about the fact this was indeed the real cause of its behavior). My configuration seems so basic to me that I can't believe I'm the only doing such a thing and I can't understand such a classical thing is not working for me. I even tried to reproduce the issue with my 3.4 tree and couldn't.
Then I guess I'll have to try 3.4. That may solve another issue I have, that may be related. I have several webservers whose logs are transfered and aggregated on one central box in the same file. I did not investigate enough to make a full report, but the central log file is corrupted (different lines seem to mix!) and it seems to me that's related to the previous fact since it seems to happen only to files that are reread from the start.I thought it may happen because syslog-ng would catch up to the end of the file with a bulk transfert and switch to a line by line transfert afterwards, or because one line fits on a buffer and not a whole file. Because logrotation happens at the same time on all webservers, all bulk transfers reach the central box at the same time. They are then written as they come and are interleaved. Because some lines overlap between 2 buffers on one webserver, if a buffer from another one is written in the middle, we have a corrupted log file. Is it possible ? Thanks for the help!
Francois Durand <fdur559@gmail.com> writes:
Then I guess I'll have to try 3.4.
I would suggest 3.3 instead (particularly, git head of it, or 3.3.6 which should be out in a week), 3.4 is at the moment, a development tree, while the 3.3 branch is stable, and aimed at production use.
That may solve another issue I have, that may be related. I have several webservers whose logs are transfered and aggregated on one central box in the same file. I did not investigate enough to make a full report, but the central log file is corrupted (different lines seem to mix!) and it seems to me that's related to the previous fact since it seems to happen only to files that are reread from the start.
Do you have a single file destination, to which these logs are routed to on the central server, or different destinations with the same file set? -- |8]
On Wed, Jul 4, 2012 at 10:54 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Francois Durand <fdur559@gmail.com> writes:
Then I guess I'll have to try 3.4.
I would suggest 3.3 instead (particularly, git head of it, or 3.3.6 which should be out in a week), 3.4 is at the moment, a development tree, while the 3.3 branch is stable, and aimed at production use.
ok, thanks for the hint!
That may solve another issue I have, that may be related. I have several webservers whose logs are transfered and aggregated on one central box in the same file. I did not investigate enough to make a full report, but the central log file is corrupted (different lines seem to mix!) and it seems to me that's related to the previous fact since it seems to happen only to files that are reread from the start.
Do you have a single file destination, to which these logs are routed to on the central server, or different destinations with the same file set?
Each webserver serves many virtual hosts. Each virtual host has one access.log. So for instance, on server 1, we have s1v1.log for vhost 1 and s1v2.log for vhost 2. On server 2, we have s2v1.log and s2v2.log. Then s1v1.log and s2v1.log are aggregated on the central log box into v1.log, and s1v2.log and s2v2.log into v2.log.
The real world case for this is when the persistent file is of the wrong version, syslog-ng will continue to run correctly, UNTIL such time as the persistent data is needed, it won't be available. syslog-ng should remove the persistent file (or rename it) and make a new one if it is out of date, or invalid for any reason. So upgrading from 2.x to 3.x (there may be other cases) and not removing that file will yield the result you are seeing. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Francois Durand [fdur559@gmail.com] Sent: Wednesday, July 04, 2012 8:31 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Followed files that grow while syslog-ng is reloaded On Wed, Jul 4, 2012 at 10:54 AM, Gergely Nagy <algernon@balabit.hu<mailto:algernon@balabit.hu>> wrote: Francois Durand <fdur559@gmail.com<mailto:fdur559@gmail.com>> writes:
Then I guess I'll have to try 3.4.
I would suggest 3.3 instead (particularly, git head of it, or 3.3.6 which should be out in a week), 3.4 is at the moment, a development tree, while the 3.3 branch is stable, and aimed at production use. ok, thanks for the hint!
That may solve another issue I have, that may be related. I have several webservers whose logs are transfered and aggregated on one central box in the same file. I did not investigate enough to make a full report, but the central log file is corrupted (different lines seem to mix!) and it seems to me that's related to the previous fact since it seems to happen only to files that are reread from the start.
Do you have a single file destination, to which these logs are routed to on the central server, or different destinations with the same file set? Each webserver serves many virtual hosts. Each virtual host has one access.log. So for instance, on server 1, we have s1v1.log for vhost 1 and s1v2.log for vhost 2. On server 2, we have s2v1.log and s2v2.log. Then s1v1.log and s2v1.log are aggregated on the central log box into v1.log, and s1v2.log and s2v2.log into v2.log.
On Wed, Jul 4, 2012 at 5:58 PM, Evan Rempel <erempel@uvic.ca> wrote:
So upgrading from 2.x to 3.x (there may be other cases) and not removing that file will yield the result you are seeing.
Thanks for the idea, but that's not the case. It's a fresh install with syslog-ng 3.2.4 since the beginning. Moreover, it seems it's working correctly for log files that receive less traffic.
participants (4)
-
Balazs Scheidler
-
Evan Rempel
-
Francois Durand
-
Gergely Nagy