<div dir="ltr">ops, maybe I was thinking about logstores, by bad. you must know better :)<br><br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div>
<br><div class="gmail_quote">On Thu, Sep 29, 2016 at 8:53 AM, Juhász, Viktor <span dir="ltr"><<a href="mailto:viktor.juhasz@balabit.com" target="_blank">viktor.juhasz@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I far as I know the disk buffer does not have journal file. Only the header of the disk buffer is mmapped.<br><div><br></div><div>BR,</div><div>Viktor Juhász</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 28, 2016 at 8:01 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Theres a journal file which is mmapped and the queue file which is a circular buffer.</p>
<p dir="ltr">But the journal is allocated in advance (a few mb), and the contents should be visible immediately as the kernel syncs it even whole syslog-ng is running.</p><div><div>
<div class="gmail_extra"><br><div class="gmail_quote">On Sep 28, 2016 12:51 PM, "Szalai, Attila" <<a href="mailto:Attila.Szalai@morganstanley.com" target="_blank">Attila.Szalai@morganstanley.c<wbr>om</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<p>
</p><div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The trick is (if I remember correctly) that even in reliable disk buffer, the content of the buffer is not _<i>flushed</i>_ to the disk with every write. It
is (and this is maybe only in my fading memories) a memory mapped file and that is the kernel responsibility to synchronize it. This is why it not safe to kernel crashes. But if you stop the syslog-ng, the content should appear in the queue.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabi<wbr>t.hu</a> [mailto:<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@list<wbr>s.balabit.hu</a>]
<b>On Behalf Of </b>Scheidler, Balázs<br>
<b>Sent:</b> Wednesday, September 28, 2016 12:48 PM<br>
<b>To:</b> Fabien Wernli; Syslog-ng users' and developers' mailing list<br>
<b>Subject:</b> Re: [syslog-ng] disk based buffering<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">log-qout-size() controls the size of the memory queue that sits next to the disk buffer, and it defaults to 64 elements. However it shouldn't apply to reliable disk buffers (e.g. reliable() setting), because
in that case everything touches the disk before syslog-ng would be sending them on.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Obviously reliable(yes) is a lot slower.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Bazsi<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <br>
Bazsi<u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal">On Wed, Sep 28, 2016 at 03:02:19AM +0200, <a href="mailto:thejaguar@tutanota.de" target="_blank">
thejaguar@tutanota.de</a> wrote:<br>
> So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not
been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable
?<br>
<br>
If I'm not mistaken the message will go to the destination's memory queue<br>
first; if the latter is full, only then will it make it to the diskq.<br>
<br>
You can check the status of the queue using the control socket<br>
(syslog-ng-ctl stats)<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">
https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">
http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<br><br>
<hr>
<br><span style="FONT-SIZE:7.5pt;FONT-FAMILY:Arial;COLOR:#808080">NOTICE:
Morgan Stanley is not acting as a municipal advisor and the opinions or views
contained herein are not intended to be, and do not constitute, advice within
the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer
Protection Act. If you have received this communication in error, please destroy
all electronic and paper copies and notify the sender immediately.
Mistransmission is not intended to waive confidentiality or privilege. Morgan
Stanley reserves the right, to the extent permitted under applicable law, to
monitor electronic communications. This message is subject to terms available at
the following link: <a href="http://www.morganstanley.com/disclaimers" target="_blank">http://www.morganstanley.com/d<wbr>isclaimers</a> If you cannot
access these links, please notify us by reply message and we will send the
contents to you. By communicating with Morgan Stanley you consent to the
foregoing and to the voice recording of conversations with personnel of Morgan
Stanley.</span><br>
<p></p>
<p></p>
<p></p>
<p></p><p></p></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div></div>
</div></div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>