[syslog-ng] disk based buffering

[Scheidler, Balázs]

ops, maybe I was thinking about logstores, by bad. you must know better :)


-- 
Bazsi

On Thu, Sep 29, 2016 at 8:53 AM, Juhász, Viktor <viktor.juhasz at balabit.com>
wrote:

> I far as I know the disk buffer does not have journal file. Only the
> header of the disk buffer is mmapped.
>
> BR,
> Viktor Juhász
>
> On Wed, Sep 28, 2016 at 8:01 PM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
>> Theres a journal file which is mmapped and the queue file which is a
>> circular buffer.
>>
>> But the journal is allocated in advance (a few mb), and the contents
>> should be visible immediately as the kernel syncs it even whole syslog-ng
>> is running.
>>
>> On Sep 28, 2016 12:51 PM, "Szalai, Attila" <Attila.Szalai at morganstanley.c
>> om> wrote:
>>
>>> The trick is (if I remember correctly) that even in reliable disk
>>> buffer, the content of the buffer is not _*flushed*_ to the disk with
>>> every write. It is (and this is maybe only in my fading memories) a memory
>>> mapped file and that is the kernel responsibility to synchronize it. This
>>> is why it not safe to kernel crashes. But if you stop the syslog-ng, the
>>> content should appear in the queue.
>>>
>>>
>>>
>>> *From:* syslog-ng-bounces at lists.balabit.hu [mailto:
>>> syslog-ng-bounces at lists.balabit.hu] *On Behalf Of *Scheidler, Balázs
>>> *Sent:* Wednesday, September 28, 2016 12:48 PM
>>> *To:* Fabien Wernli; Syslog-ng users' and developers' mailing list
>>> *Subject:* Re: [syslog-ng] disk based buffering
>>>
>>>
>>>
>>> log-qout-size() controls the size of the memory queue that sits next to
>>> the disk buffer, and it defaults to 64 elements. However it shouldn't apply
>>> to reliable disk buffers (e.g. reliable() setting), because in that case
>>> everything touches the disk before syslog-ng would be sending them on.
>>>
>>> Obviously reliable(yes) is a lot slower.
>>>
>>> Bazsi
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Bazsi
>>>
>>>
>>>
>>> On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>>>
>>> On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar at tutanota.de wrote:
>>> > So with dqtool ( cool) I noticed that actual problem is that syslog-ng
>>> is not writing to disk queue immediately when a message is generated.  The
>>> message goes to usual place like var/log/message etc but not in the queue
>>> where it should because it has not been transmitted to remote host.  Now I
>>> am thinking there has to be a memory limit before it starts
>>> flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1)
>>> and log-fifo-size(1) but none of them works. Is it a static value or is it
>>> configurable ?
>>>
>>> If I'm not mistaken the message will go to the destination's memory queue
>>> first; if the latter is full, only then will it make it to the diskq.
>>>
>>> You can check the status of the queue using the control socket
>>> (syslog-ng-ctl stats)
>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>>> opinions or views contained herein are not intended to be, and do not
>>> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
>>> Street Reform and Consumer Protection Act. If you have received this
>>> communication in error, please destroy all electronic and paper copies and
>>> notify the sender immediately. Mistransmission is not intended to waive
>>> confidentiality or privilege. Morgan Stanley reserves the right, to the
>>> extent permitted under applicable law, to monitor electronic
>>> communications. This message is subject to terms available at the following
>>> link: http://www.morganstanley.com/disclaimers  If you cannot access
>>> these links, please notify us by reply message and we will send the
>>> contents to you. By communicating with Morgan Stanley you consent to the
>>> foregoing and to the voice recording of conversations with personnel of
>>> Morgan Stanley.
>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160929/c3c8d797/attachment.htm