[syslog-ng] disk based buffering

[Juhász, Viktor]

I far as I know the disk buffer does not have journal file. Only the header
of the disk buffer is mmapped.

BR,
Viktor Juhász

On Wed, Sep 28, 2016 at 8:01 PM, Scheidler, Balázs <
balazs.scheidler at balabit.com> wrote:

> Theres a journal file which is mmapped and the queue file which is a
> circular buffer.
>
> But the journal is allocated in advance (a few mb), and the contents
> should be visible immediately as the kernel syncs it even whole syslog-ng
> is running.
>
> On Sep 28, 2016 12:51 PM, "Szalai, Attila" <Attila.Szalai at morganstanley.
> com> wrote:
>
>> The trick is (if I remember correctly) that even in reliable disk buffer,
>> the content of the buffer is not _*flushed*_ to the disk with every
>> write. It is (and this is maybe only in my fading memories) a memory mapped
>> file and that is the kernel responsibility to synchronize it. This is why
>> it not safe to kernel crashes. But if you stop the syslog-ng, the content
>> should appear in the queue.
>>
>>
>>
>> *From:* syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at list
>> s.balabit.hu] *On Behalf Of *Scheidler, Balázs
>> *Sent:* Wednesday, September 28, 2016 12:48 PM
>> *To:* Fabien Wernli; Syslog-ng users' and developers' mailing list
>> *Subject:* Re: [syslog-ng] disk based buffering
>>
>>
>>
>> log-qout-size() controls the size of the memory queue that sits next to
>> the disk buffer, and it defaults to 64 elements. However it shouldn't apply
>> to reliable disk buffers (e.g. reliable() setting), because in that case
>> everything touches the disk before syslog-ng would be sending them on.
>>
>> Obviously reliable(yes) is a lot slower.
>>
>> Bazsi
>>
>>
>>
>>
>>
>>
>> --
>> Bazsi
>>
>>
>>
>> On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>>
>> On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar at tutanota.de wrote:
>> > So with dqtool ( cool) I noticed that actual problem is that syslog-ng
>> is not writing to disk queue immediately when a message is generated.  The
>> message goes to usual place like var/log/message etc but not in the queue
>> where it should because it has not been transmitted to remote host.  Now I
>> am thinking there has to be a memory limit before it starts
>> flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1)
>> and log-fifo-size(1) but none of them works. Is it a static value or is it
>> configurable ?
>>
>> If I'm not mistaken the message will go to the destination's memory queue
>> first; if the latter is full, only then will it make it to the diskq.
>>
>> You can check the status of the queue using the control socket
>> (syslog-ng-ctl stats)
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>>
>> ------------------------------
>>
>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> opinions or views contained herein are not intended to be, and do not
>> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
>> Street Reform and Consumer Protection Act. If you have received this
>> communication in error, please destroy all electronic and paper copies and
>> notify the sender immediately. Mistransmission is not intended to waive
>> confidentiality or privilege. Morgan Stanley reserves the right, to the
>> extent permitted under applicable law, to monitor electronic
>> communications. This message is subject to terms available at the following
>> link: http://www.morganstanley.com/disclaimers  If you cannot access
>> these links, please notify us by reply message and we will send the
>> contents to you. By communicating with Morgan Stanley you consent to the
>> foregoing and to the voice recording of conversations with personnel of
>> Morgan Stanley.
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160929/d3948aeb/attachment-0001.htm