[syslog-ng] Log rotate without log rotate.

[Bruno Vieira]

Hi Justin.

Thank you for your help. In the project I'm in, I won't have the time to do
it in a low level (kernel) as you refer and as I also thought it would be
great if Linux already had a device like this (a rotation file with
parameters of lines and/or size).

So, my solution will be something like a buffer in a daemon and for each new
line, I can do a shift on the buffer. I will implement a behaviour more or
less like this.

I like your suggestion of the linked lists, but a buffer will also do the
way since I never want to change the order.. I mean, no need to change the
pointers NEXT and and PREVIOUS. A simple shift of elements will make sense..

Many regards for your so complete solution.
Bruno.

On 3/27/07, Justin Randall <jrandall at comwave.net> wrote:
>
>  In fact, you could even write this as a kernel module (which would give
> you direct access to the list.h kernel header) and implement this
> behaviour right under the hood of the OS rather than in Syslog-NG.
>
>
>
> Basically, using the Linux VFS and kernel module subsystems, you can
> create device files in which the "write" operations behave in the manner
> listed below (writing to a circular list implementation of X number of
> nodes).  This would then allow Syslog-NG to continuously write to the device
> file, and let the device file's implementation operate in the ring buffer
> manner outlined below.
>
>
>
> I hope this helps,
>
>
>
> Justin.
>
>
>  ------------------------------
>
> *From:* Justin Randall
> *Sent:* Tuesday, March 27, 2007 5:47 PM
> *To:* 'Syslog-ng users' and developers' mailing list'
> *Subject:* RE: [syslog-ng] Log rotate without log rotate.
>
>
>
> Depending on the language you are using, if you have access to pointers
> (if you are using C/C++) you can very efficiently implement this behaviour
> using a circular ring buffer.  Essentially this would involve a
> double-linked list implementation in which the head node's "prev" member
> references the tail node of the list.
>
>
>
> The list implementation used in the Linux kernel (list.h in the kernel
> header files) uses this type of list.  Using this list implementation, you
> can have your circular buffer by doing the following:
>
>
>
> -          Create a list with 100 nodes, each node representing a "line"
> of logging text.
>
> -          Create an integrator-type pointer used to traverse the list.
>
> -          Each node contains a "next" and a "prev" pointer which
> reference the next or previous node in the list.
>
> -          When writing a line, modify the text line for the node
> currently referenced by the integrator pointer, and the advance the iterator
> to the value of the "next" pointer for the current node.
>
>
>
> Using this logic, the middle-application will be very compact, as it does
> not need any program logic to determine if it is in need to dump old data
> for new data.
>
>
>
> You may have already gone down this path, and if so I apologize for the
> long-winded email, though I hope this is helpful in writing a light-weight
> middle-app.
>
>
>
> Regards,
>
>
>
> Justin.
>
>
>  ------------------------------
>
> *From:* Bruno Vieira [mailto:giuliapo at gmail.com]
> *Sent:* Saturday, March 17, 2007 12:58 PM
> *To:* Syslog-ng users' and developers' mailing list
> *Subject:* Re: [syslog-ng] Log rotate without log rotate.
>
>
>
> Hi there. Thank you for your opinion.
>
> I have found that this really is a feature well needed by many
> embeddedsystems.
>
> I will implement or reuse a component that makes this kind of rotation...
> the conclusion i got was that what I really needed was a file that would
> implement a buffer... so, the only way is making a daemon that does this for
> me.. and it may or may not save it's buffer on a destination file.
>
> It would be great if Linux had a driver that would do this on it's own. Or
> at least a driver on the project openwrt (linux for embedded systems).
>
> Well, I have the solution now :)
>
> Thanks you all for your contribution.
> Many regards,
> Bruno.
>
>
>
>  On 17 Mar 2007 16:25:32 +0000, *Bryan Henderson* <bryanh at giraffe-data.com
> > wrote:
>
> >I have a solution.. but I think there can be a better one. My solution
> is:
> >having a midlle program between syslog-ng and the destination file. And
> for
> >every message received, the program would verify if the destination file
> >(p.e. auth.log) already has 100 lines. If so, deleted the last line, make
>
> >all others one ENTER down and that append the new line to the beginning
> of
> >the file.
>
> It would probably be better to maintain the file in circular fashion:
> allocate 8K in the beginning, then write messages from beginning to
> end, then wrap around and write from the beginning again, always
> keeping track (in a header of the file) of where you last wrote.  Use
> a special program that understands this file format to translate it to
> a regular stream that you can use normally.
>
> Lots of systems have programs that do this.  I looked on Freshmeat for
> a publicly available one and found 'cupyvei', though I don't much care
> for the details of this implementation.  It would be easy to write a
> program of your own.
>
> I also think a built in log destination of this type
> (e.g. "file_circular") would be an excellent addition to syslog-ng.  I
> think this type of logging is a common requirement.
>
> --
> Bryan Henderson                                   San Jose, California
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>
>
>
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070328/9b5ff581/attachment.htm