Re: [syslog-ng] log file size limit
On 2011-07-25 17:50, William Tambellini wrote:
Dear Syslog-ng team,
We are currently evaluating syslog-ng in order to satisfy some requests from our customers regarding logs from our software. One of their requests is to limit log file size using some kind of rotation algo : if size > 6Mo then rotate to another file. We have found some options in file destination like fifo buffer control but nothing about files rotation. Is there anything for that kind of feature ?
I have a patch that does something of that kind. When the log file grows above certain size syslog-ng switches to a new one and renames the old one. Further processing (such as compression) should be done externally. I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed... -- Sergei.
On Mon, Jul 25, 2011 at 19:58, Sergei Zhirikov <sfzhi@yahoo.com> wrote:
On 2011-07-25 17:50, William Tambellini wrote:
Dear Syslog-ng team,
We are currently evaluating syslog-ng in order to satisfy some requests from our customers regarding logs from our software. One of their requests is to limit log file size using some kind of rotation algo : if size > 6Mo then rotate to another file. We have found some options in file destination like fifo buffer control but nothing about files rotation. Is there anything for that kind of feature ?
I have a patch that does something of that kind. When the log file grows above certain size syslog-ng switches to a new one and renames the old one. Further processing (such as compression) should be done externally.
I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed...
Can you please post it again? I'd love to have a look. -- |8]
On 2011-07-25 20:03, Gergely Nagy wrote:
On Mon, Jul 25, 2011 at 19:58, Sergei Zhirikov<sfzhi@yahoo.com> wrote:
On 2011-07-25 17:50, William Tambellini wrote:
Dear Syslog-ng team,
We are currently evaluating syslog-ng in order to satisfy some requests from our customers regarding logs from our software. One of their requests is to limit log file size using some kind of rotation algo : if size> 6Mo then rotate to another file. We have found some options in file destination like fifo buffer control but nothing about files rotation. Is there anything for that kind of feature ?
I have a patch that does something of that kind. When the log file grows above certain size syslog-ng switches to a new one and renames the old one. Further processing (such as compression) should be done externally.
I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed...
Can you please post it again? I'd love to have a look.
Attached. The patch is against syslog-ng-3.1.4, but the idea is pretty straight forward, so I don't expect it to be difficult to port to a newer version. The patch introduces two new configuration options: 'file_size_limit' to be used inside global 'options' and 'size_limit' to be used inside 'file' destination. Each option specifies log file size limit in bytes. If the global option is set to a value greater than zero it applies to all 'file' destinations. A particular file destination can remove the limit by setting it to zero. For example: # set the global file size limit options { file_size_limit(123456); }; # set a different size limit for a particular file destination destination log1 { file("/var/log/log1.log" size_limit(456789)); }; # remove size limit for a particular file destination # (useful only if there is a global size limit set) destination log2 { file("/var/log/log2.log" size_limit(0)); }; The file size is checked after writing each log message and if the file has grown up to or above the size limit the file is renamed and a new empty file is created to continue logging to. The name format the "overgrown" log file is renamed to is "<p>-<s>.<m>-<r>", where <p> - the full path and name of the original log file, <s> - current time in UNIX format (seconds since Jan 1, 1970), <m> - fractional part of the current time (microseconds, 6 digits), <r> - a random number (10 digits). The intended use is to have incrond or another similar mechanism to detect when there is a new "renamed" log file and to process it in whatever way necessary (gzip it, parse it, send it my email, etc.). -- Sergei.
Sergei Zhirikov <sfzhi@yahoo.com> writes:
I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed...
Can you please post it again? I'd love to have a look.
Attached.
Thank you, I will have a look hopefully within a day or two! (Judging by the description, it sounds very useful & promising to me!) -- |8]
On Mon, 2011-07-25 at 20:46 +0200, Gergely Nagy wrote:
Sergei Zhirikov <sfzhi@yahoo.com> writes:
I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed...
Can you please post it again? I'd love to have a look.
Attached.
Thank you, I will have a look hopefully within a day or two!
(Judging by the description, it sounds very useful & promising to me!)
any news on this? -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
On Mon, 2011-07-25 at 20:46 +0200, Gergely Nagy wrote:
Sergei Zhirikov <sfzhi@yahoo.com> writes:
I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed...
Can you please post it again? I'd love to have a look.
Attached.
Thank you, I will have a look hopefully within a day or two!
(Judging by the description, it sounds very useful & promising to me!)
any news on this?
Not yet. Got caught up in DebConf stuff, and still not recovered. I'll have a go at it this coming friday. -- |8]
Gergely Nagy <algernon@balabit.hu> writes:
Balazs Scheidler <bazsi@balabit.hu> writes:
On Mon, 2011-07-25 at 20:46 +0200, Gergely Nagy wrote:
Sergei Zhirikov <sfzhi@yahoo.com> writes:
I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed...
Can you please post it again? I'd love to have a look.
Attached.
Thank you, I will have a look hopefully within a day or two!
(Judging by the description, it sounds very useful & promising to me!)
any news on this?
Not yet. Got caught up in DebConf stuff, and still not recovered. I'll have a go at it this coming friday.
Update: I'm halfway through. Applied the patch, made it compile, figured out why it doesn't work (it modified LogTransportPlain to have a callback, but nowadays we're using LogProtoFileWriter to write to files, so the callback never gets called. My first idea would be to port the patch to extend LogProtoFileWriter instead of LogTransportPlain, but I'm not done figuring out how that part of the code works, yet. In any case, the patch itself looks good, there's only one thing I will want to change after porting it to 3.3: I don't like how the renamed file has a hard-coded template. I'd much rather introduce a second, optional argument to size_limit(), which would be the template to rename the old file to (defaulting to something like the current template, which appends a timestamp and a random number). However, adding such a template will need a bit more thinking, as I'll need to pass down the neccessary structures to the callback. Or perhaps, the callback could be rewritten to not do the actual file reopening, just determine if we need to do that. And if so, log_proto_writer_flush() would bubble up, we'd catch it somewhere in affile, notice a flag the callback would set, and do the reopen, and retry. This way we have easy access to the message, and the callback would be very lightweight. I'll see what I can do about this. -- |8]
On Fri, 2011-08-19 at 12:01 +0200, Gergely Nagy wrote:
Gergely Nagy <algernon@balabit.hu> writes:
Balazs Scheidler <bazsi@balabit.hu> writes:
On Mon, 2011-07-25 at 20:46 +0200, Gergely Nagy wrote:
Sergei Zhirikov <sfzhi@yahoo.com> writes:
> I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed...
Can you please post it again? I'd love to have a look.
Attached.
Thank you, I will have a look hopefully within a day or two!
(Judging by the description, it sounds very useful & promising to me!)
any news on this?
Not yet. Got caught up in DebConf stuff, and still not recovered. I'll have a go at it this coming friday.
Update: I'm halfway through. Applied the patch, made it compile, figured out why it doesn't work (it modified LogTransportPlain to have a callback, but nowadays we're using LogProtoFileWriter to write to files, so the callback never gets called.
My first idea would be to port the patch to extend LogProtoFileWriter instead of LogTransportPlain, but I'm not done figuring out how that part of the code works, yet.
In any case, the patch itself looks good, there's only one thing I will want to change after porting it to 3.3: I don't like how the renamed file has a hard-coded template. I'd much rather introduce a second, optional argument to size_limit(), which would be the template to rename the old file to (defaulting to something like the current template, which appends a timestamp and a random number).
However, adding such a template will need a bit more thinking, as I'll need to pass down the neccessary structures to the callback.
Or perhaps, the callback could be rewritten to not do the actual file reopening, just determine if we need to do that. And if so, log_proto_writer_flush() would bubble up, we'd catch it somewhere in affile, notice a flag the callback would set, and do the reopen, and retry.
This way we have easy access to the message, and the callback would be very lightweight.
I'll see what I can do about this.
I've reviewed this code and I have some open issues, like: 1) the LogProtoFileWriter is only used by the file destination, no need to use a separate constructor. I think it'd make sense to move the FileWriter proto class into the file plugin. 2) I don't like the global option. For one I doubt that a global size limit would ever really be used: if they are really multiple destination files, they should probably have different size limits. 3) The sequence number of renaming should probably be persistent, as reloads/restarts will trigger that counter to restart from 1. And I don't really like to overload the $SEQNUM macro which is defined for a completely different purpose. Otherwise I do like the functionality, but I don't really have an answer for these questions either. The code could probably be made nicer, simply be not pretending that LogProtoFileWriter is generic code. It is not, it belongs to the file plugin. The SEQNUM stuff is a good idea (I really couldn't decipher it first, but then I liked it), however it can be confusing. Certainly I'd need some thinking time, but I thought I'd post these concerns anyway, to see that I'm not just idling on integrating it. -- Bazsi
participants (4)
-
Balazs Scheidler
-
Gergely Nagy
-
Gergely Nagy
-
Sergei Zhirikov