[syslog-ng] disk based buffering

[Scheidler, Balázs]

Theres a journal file which is mmapped and the queue file which is a
circular buffer.

But the journal is allocated in advance (a few mb), and the contents should
be visible immediately as the kernel syncs it even whole syslog-ng is
running.

On Sep 28, 2016 12:51 PM, "Szalai, Attila" <Attila.Szalai at morganstanley.com>
wrote:

> The trick is (if I remember correctly) that even in reliable disk buffer,
> the content of the buffer is not _*flushed*_ to the disk with every
> write. It is (and this is maybe only in my fading memories) a memory mapped
> file and that is the kernel responsibility to synchronize it. This is why
> it not safe to kernel crashes. But if you stop the syslog-ng, the content
> should appear in the queue.
>
>
>
> *From:* syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces@
> lists.balabit.hu] *On Behalf Of *Scheidler, Balázs
> *Sent:* Wednesday, September 28, 2016 12:48 PM
> *To:* Fabien Wernli; Syslog-ng users' and developers' mailing list
> *Subject:* Re: [syslog-ng] disk based buffering
>
>
>
> log-qout-size() controls the size of the memory queue that sits next to
> the disk buffer, and it defaults to 64 elements. However it shouldn't apply
> to reliable disk buffers (e.g. reliable() setting), because in that case
> everything touches the disk before syslog-ng would be sending them on.
>
> Obviously reliable(yes) is a lot slower.
>
> Bazsi
>
>
>
>
>
>
> --
> Bazsi
>
>
>
> On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>
> On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar at tutanota.de wrote:
> > So with dqtool ( cool) I noticed that actual problem is that syslog-ng
> is not writing to disk queue immediately when a message is generated.  The
> message goes to usual place like var/log/message etc but not in the queue
> where it should because it has not been transmitted to remote host.  Now I
> am thinking there has to be a memory limit before it starts
> flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1)
> and log-fifo-size(1) but none of them works. Is it a static value or is it
> configurable ?
>
> If I'm not mistaken the message will go to the destination's memory queue
> first; if the latter is full, only then will it make it to the diskq.
>
> You can check the status of the queue using the control socket
> (syslog-ng-ctl stats)
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies and
> notify the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to the
> extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following
> link: http://www.morganstanley.com/disclaimers  If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By communicating with Morgan Stanley you consent to the
> foregoing and to the voice recording of conversations with personnel of
> Morgan Stanley.
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160928/c9c79a66/attachment.htm