New subject: Log rotate without log rotate.

27 Mar 2007

      In fact, you could even write this as a kernel module (which would give
you direct access to the list.h kernel header) and implement this
behaviour right under the hood of the OS rather than in Syslog-NG.

Basically, using the Linux VFS and kernel module subsystems, you can
create device files in which the "write" operations behave in the manner
listed below (writing to a circular list implementation of X number of
nodes).  This would then allow Syslog-NG to continuously write to the
device file, and let the device file's implementation operate in the
ring buffer manner outlined below.

I hope this helps,

Justin.

  _____  

From: Justin Randall 
Sent: Tuesday, March 27, 2007 5:47 PM
To: 'Syslog-ng users' and developers' mailing list'
Subject: RE: [syslog-ng] Log rotate without log rotate.

Depending on the language you are using, if you have access to pointers
(if you are using C/C++) you can very efficiently implement this
behaviour using a circular ring buffer.  Essentially this would involve
a double-linked list implementation in which the head node's "prev"
member references the tail node of the list.

The list implementation used in the Linux kernel (list.h in the kernel
header files) uses this type of list.  Using this list implementation,
you can have your circular buffer by doing the following:

-          Create a list with 100 nodes, each node representing a "line"
of logging text.

-          Create an integrator-type pointer used to traverse the list.

-          Each node contains a "next" and a "prev" pointer which
reference the next or previous node in the list.

-          When writing a line, modify the text line for the node
currently referenced by the integrator pointer, and the advance the
iterator to the value of the "next" pointer for the current node.

Using this logic, the middle-application will be very compact, as it
does not need any program logic to determine if it is in need to dump
old data for new data.

You may have already gone down this path, and if so I apologize for the
long-winded email, though I hope this is helpful in writing a
light-weight middle-app.

Regards,

Justin.

  _____  

From: Bruno Vieira [mailto:giuliapo@gmail.com] 
Sent: Saturday, March 17, 2007 12:58 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Log rotate without log rotate.

Hi there. Thank you for your opinion.

I have found that this really is a feature well needed by many
embeddedsystems.

I will implement or reuse a component that makes this kind of
rotation... the conclusion i got was that what I really needed was a
file that would implement a buffer... so, the only way is making a
daemon that does this for me.. and it may or may not save it's buffer on
a destination file. 

It would be great if Linux had a driver that would do this on it's own.
Or at least a driver on the project openwrt (linux for embedded
systems).

Well, I have the solution now :)

Thanks you all for your contribution. 
Many regards,
Bruno.

On 17 Mar 2007 16:25:32 +0000, Bryan Henderson <bryanh@giraffe-data.com
...
wrote:
...
I have a solution.. but I think there can be a better one. My solution
is: 
having a midlle program between syslog-ng and the destination file. And
for
every message received, the program would verify if the destination
file
(p.e. auth.log) already has 100 lines. If so, deleted the last line,
make 
all others one ENTER down and that append the new line to the beginning
of
the file.
It would probably be better to maintain the file in circular fashion:
allocate 8K in the beginning, then write messages from beginning to 
end, then wrap around and write from the beginning again, always
keeping track (in a header of the file) of where you last wrote.  Use
a special program that understands this file format to translate it to
a regular stream that you can use normally. 

Lots of systems have programs that do this.  I looked on Freshmeat for
a publicly available one and found 'cupyvei', though I don't much care
for the details of this implementation.  It would be easy to write a 
program of your own.

I also think a built in log destination of this type
(e.g. "file_circular") would be an excellent addition to syslog-ng.  I
think this type of logging is a common requirement. 

--
Bryan Henderson                                   San Jose, California
_______________________________________________
syslog-ng maillist  -  syslog-ng@lists.balabit.hu 
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

RE: [syslog-ng] Log rotate without log rotate.

Justin Randall

Bruno Vieira

tags

participants (2)