Log rotate without log rotate.
Hi there. I have a question and at the same time i almost know that it must exist a way of doing this. The situation is: I want that my logs act as FIFOs. I mean.. I want my log files to have a max of 100 KB or 100 lines (the one that is most convenient to find a solution). If I use a pipe, I have the problem that if I read on one side of the pipe, it continues dumping it's content until getting empty regardless of having more messages coming in through the other end. But what I want is a pipe that that keeps it's content and only flushes a message out every time it receives a message in. Just like a FIFO.. I want this behaviour in a file... named pipe? I don't know the solution for this but I can feel that it is out there :) This way I would have a log rotation without using log rotate that normally it copies the file content to a new one and than syslog-ng starts writing on an empty one. Tell me what you think about this. Best regards, Bruno.
Hi, Bruno Vieira <giuliapo@gmail.com> [20070316 14:59:00 +0000]:
Hi there.
I have a question and at the same time i almost know that it must exist a way of doing this.
[snipped]
This way I would have a log rotation without using log rotate that normally it copies the file content to a new one and than syslog-ng starts writing on an empty one.
I might have missed something but whats wrong with something like: ==== destination d_cheese { file("/var/log/cheese/$R_YEAR$R_MONTH$R_DAY.log" perm(0644)); }; ==== Have a look at the follow macro expansions for more fun: http://www.balabit.com/products/syslog_ng/reference-2.0/syslog-ng.html/index...
Tell me what you think about this.
shear craziness :) Cheers Alex
Hi Alex. Thanks you for your considerations. But the problem here is that the device that will be logging have memory restrictions. So, I want a log that acts as a fifo. And every time a new message comes, the oldest one gets out. With log rotate I can't guarantee that the size of the log keeps in a acceptable interval like 100 - 120 KB. Because if I get a ton of info dumped to the log file within a small amount of time, I'll still overshoot the maximum file size. I would like to have only one file per destination (ex: auth.log) and keep that file within 100 - 120 KB or 100 lines, per example. I want it to be circular... to be like a FIFO. What do you think? I have a solution.. but I think there can be a better one. My solution is: having a midlle program between syslog-ng and the destination file. And for every message received, the program would verify if the destination file ( p.e. auth.log) already has 100 lines. If so, deleted the last line, make all others one ENTER down and that append the new line to the beginning of the file. But this seems to be a very "manual" solution... Best regards, Bruno. ====
destination d_cheese { file("/var/log/cheese/$R_YEAR$R_MONTH$R_DAY.log" perm(0644)); }; ====
Have a look at the follow macro expansions for more fun:
http://www.balabit.com/products/syslog_ng/reference-2.0/syslog-ng.html/index...
Tell me what you think about this.
shear craziness :)
Cheers
Alex _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
I have a solution.. but I think there can be a better one. My solution is: having a midlle program between syslog-ng and the destination file. And for every message received, the program would verify if the destination file (p.e. auth.log) already has 100 lines. If so, deleted the last line, make all others one ENTER down and that append the new line to the beginning of the file.
It would probably be better to maintain the file in circular fashion: allocate 8K in the beginning, then write messages from beginning to end, then wrap around and write from the beginning again, always keeping track (in a header of the file) of where you last wrote. Use a special program that understands this file format to translate it to a regular stream that you can use normally. Lots of systems have programs that do this. I looked on Freshmeat for a publicly available one and found 'cupyvei', though I don't much care for the details of this implementation. It would be easy to write a program of your own. I also think a built in log destination of this type (e.g. "file_circular") would be an excellent addition to syslog-ng. I think this type of logging is a common requirement. -- Bryan Henderson San Jose, California
Hi there. Thank you for your opinion. I have found that this really is a feature well needed by many embeddedsystems. I will implement or reuse a component that makes this kind of rotation... the conclusion i got was that what I really needed was a file that would implement a buffer... so, the only way is making a daemon that does this for me.. and it may or may not save it's buffer on a destination file. It would be great if Linux had a driver that would do this on it's own. Or at least a driver on the project openwrt (linux for embedded systems). Well, I have the solution now :) Thanks you all for your contribution. Many regards, Bruno. On 17 Mar 2007 16:25:32 +0000, Bryan Henderson <bryanh@giraffe-data.com> wrote:
I have a solution.. but I think there can be a better one. My solution is: having a midlle program between syslog-ng and the destination file. And for every message received, the program would verify if the destination file (p.e. auth.log) already has 100 lines. If so, deleted the last line, make all others one ENTER down and that append the new line to the beginning of the file.
It would probably be better to maintain the file in circular fashion: allocate 8K in the beginning, then write messages from beginning to end, then wrap around and write from the beginning again, always keeping track (in a header of the file) of where you last wrote. Use a special program that understands this file format to translate it to a regular stream that you can use normally.
Lots of systems have programs that do this. I looked on Freshmeat for a publicly available one and found 'cupyvei', though I don't much care for the details of this implementation. It would be easy to write a program of your own.
I also think a built in log destination of this type (e.g. "file_circular") would be an excellent addition to syslog-ng. I think this type of logging is a common requirement.
-- Bryan Henderson San Jose, California _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Fri, 16 Mar 2007 14:59:00 -0000, Bruno Vieira said:
I have a question and at the same time i almost know that it must exist a way of doing this.
The situation is: I want that my logs act as FIFOs. I mean.. I want my log files to have a max of 100 KB or 100 lines (the one that is most convenient to find a solution).
Trust me - you *really* want to break it on the basis of discrete times that tend to have 100K per time period (be it once an hour, or once a day, or every 15 minutes, or something). When you're trying to shoot a problem, and one logfile destination runs from 02:17:34 to 03:09:19, the next from 03:09:19 to 05:48:28, and another destination runs from 01:45:07 to 02:36:12, the next from 02:36:12 to 06:29:58 - quick, which files do you need to get all messages from 45 seconds to either side of 03:10:15? Oh, and one of the machines concerned didn't have the DST patch installed, so you probably want to find 45 seconds either side of 02:10:15 as well. ;)
participants (4)
-
Alexander Clouter
-
Bruno Vieira
-
bryanh＠giraffe-data.com
-
Valdis.Kletnieks＠vt.edu