Re: [syslog-ng] disk based buffering
So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ? Thanks 27. Sep 2016 12:41 by thejaguar@tutanota.de:
Thanks.> So if the queue stays intact, syslog-ng will try to send unsent messages as and when it starts ? even after 2-3 days ? it does not reset the queue or tracking ever ? Thanks again
27. Sep 2016 12:11 by > balazs.scheidler@balabit.com> :
Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact.
There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package.
On Sep 27, 2016 8:35 PM, <>> thejaguar@tutanota.de>> > wrote:
>>> Hi,>>> I have been using disk based buffering with reliable turned on yes as suggested here :->>> https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o...
This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?>>> I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? =======================
destination d_net {>>> network (>>> "`myloghost`" port(`mylogport`) transport("tls")>>> tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) )>>> disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) )>>> template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n")>>> );>>> }; syslog-ng 3.8.1>>> Installer-Version: 3.8.1>>> Revision:>>> Module-Directory: /usr/lib/syslog-ng>>> Module-Path: /usr/lib/syslog-ng>>> Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsot>>> Enable-Debug: off>>> Enable-GProf: off>>> Enable-Memtrace: off>>> Enable-IPv6: off>>> Enable-Spoof-Source: off>>> Enable-TCP-Wrapper: off>>> Enable-Linux-Caps: off =======================
Thanks
______________________________________________________________________________ Member info: >>> https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: >>> http://www.balabit.com/wiki/syslog-ng-faq
On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar@tutanota.de wrote:
So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ?
If I'm not mistaken the message will go to the destination's memory queue first; if the latter is full, only then will it make it to the diskq. You can check the status of the queue using the control socket (syslog-ng-ctl stats)
log-qout-size() controls the size of the memory queue that sits next to the disk buffer, and it defaults to 64 elements. However it shouldn't apply to reliable disk buffers (e.g. reliable() setting), because in that case everything touches the disk before syslog-ng would be sending them on. Obviously reliable(yes) is a lot slower. Bazsi -- Bazsi On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar@tutanota.de wrote:
So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ?
If I'm not mistaken the message will go to the destination's memory queue first; if the latter is full, only then will it make it to the diskq.
You can check the status of the queue using the control socket (syslog-ng-ctl stats)
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The trick is (if I remember correctly) that even in reliable disk buffer, the content of the buffer is not _flushed_ to the disk with every write. It is (and this is maybe only in my fading memories) a memory mapped file and that is the kernel responsibility to synchronize it. This is why it not safe to kernel crashes. But if you stop the syslog-ng, the content should appear in the queue. From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scheidler, Balázs Sent: Wednesday, September 28, 2016 12:48 PM To: Fabien Wernli; Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] disk based buffering log-qout-size() controls the size of the memory queue that sits next to the disk buffer, and it defaults to 64 elements. However it shouldn't apply to reliable disk buffers (e.g. reliable() setting), because in that case everything touches the disk before syslog-ng would be sending them on. Obviously reliable(yes) is a lot slower. Bazsi -- Bazsi On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli@in2p3.fr<mailto:wernli@in2p3.fr>> wrote: On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar@tutanota.de<mailto:thejaguar@tutanota.de> wrote:
So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ?
If I'm not mistaken the message will go to the destination's memory queue first; if the latter is full, only then will it make it to the diskq. You can check the status of the queue using the control socket (syslog-ng-ctl stats) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.
Theres a journal file which is mmapped and the queue file which is a circular buffer. But the journal is allocated in advance (a few mb), and the contents should be visible immediately as the kernel syncs it even whole syslog-ng is running. On Sep 28, 2016 12:51 PM, "Szalai, Attila" <Attila.Szalai@morganstanley.com> wrote:
The trick is (if I remember correctly) that even in reliable disk buffer, the content of the buffer is not _*flushed*_ to the disk with every write. It is (and this is maybe only in my fading memories) a memory mapped file and that is the kernel responsibility to synchronize it. This is why it not safe to kernel crashes. But if you stop the syslog-ng, the content should appear in the queue.
*From:* syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@ lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Wednesday, September 28, 2016 12:48 PM *To:* Fabien Wernli; Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] disk based buffering
log-qout-size() controls the size of the memory queue that sits next to the disk buffer, and it defaults to 64 elements. However it shouldn't apply to reliable disk buffers (e.g. reliable() setting), because in that case everything touches the disk before syslog-ng would be sending them on.
Obviously reliable(yes) is a lot slower.
Bazsi
-- Bazsi
On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar@tutanota.de wrote:
So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ?
If I'm not mistaken the message will go to the destination's memory queue first; if the latter is full, only then will it make it to the diskq.
You can check the status of the queue using the control socket (syslog-ng-ctl stats)
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I far as I know the disk buffer does not have journal file. Only the header of the disk buffer is mmapped. BR, Viktor Juhász On Wed, Sep 28, 2016 at 8:01 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Theres a journal file which is mmapped and the queue file which is a circular buffer.
But the journal is allocated in advance (a few mb), and the contents should be visible immediately as the kernel syncs it even whole syslog-ng is running.
On Sep 28, 2016 12:51 PM, "Szalai, Attila" <Attila.Szalai@morganstanley. com> wrote:
The trick is (if I remember correctly) that even in reliable disk buffer, the content of the buffer is not _*flushed*_ to the disk with every write. It is (and this is maybe only in my fading memories) a memory mapped file and that is the kernel responsibility to synchronize it. This is why it not safe to kernel crashes. But if you stop the syslog-ng, the content should appear in the queue.
*From:* syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@list s.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Wednesday, September 28, 2016 12:48 PM *To:* Fabien Wernli; Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] disk based buffering
log-qout-size() controls the size of the memory queue that sits next to the disk buffer, and it defaults to 64 elements. However it shouldn't apply to reliable disk buffers (e.g. reliable() setting), because in that case everything touches the disk before syslog-ng would be sending them on.
Obviously reliable(yes) is a lot slower.
Bazsi
-- Bazsi
On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar@tutanota.de wrote:
So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ?
If I'm not mistaken the message will go to the destination's memory queue first; if the latter is full, only then will it make it to the diskq.
You can check the status of the queue using the control socket (syslog-ng-ctl stats)
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
ops, maybe I was thinking about logstores, by bad. you must know better :) -- Bazsi On Thu, Sep 29, 2016 at 8:53 AM, Juhász, Viktor <viktor.juhasz@balabit.com> wrote:
I far as I know the disk buffer does not have journal file. Only the header of the disk buffer is mmapped.
BR, Viktor Juhász
On Wed, Sep 28, 2016 at 8:01 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Theres a journal file which is mmapped and the queue file which is a circular buffer.
But the journal is allocated in advance (a few mb), and the contents should be visible immediately as the kernel syncs it even whole syslog-ng is running.
On Sep 28, 2016 12:51 PM, "Szalai, Attila" <Attila.Szalai@morganstanley.c om> wrote:
The trick is (if I remember correctly) that even in reliable disk buffer, the content of the buffer is not _*flushed*_ to the disk with every write. It is (and this is maybe only in my fading memories) a memory mapped file and that is the kernel responsibility to synchronize it. This is why it not safe to kernel crashes. But if you stop the syslog-ng, the content should appear in the queue.
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Wednesday, September 28, 2016 12:48 PM *To:* Fabien Wernli; Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] disk based buffering
log-qout-size() controls the size of the memory queue that sits next to the disk buffer, and it defaults to 64 elements. However it shouldn't apply to reliable disk buffers (e.g. reliable() setting), because in that case everything touches the disk before syslog-ng would be sending them on.
Obviously reliable(yes) is a lot slower.
Bazsi
-- Bazsi
On Wed, Sep 28, 2016 at 9:02 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Wed, Sep 28, 2016 at 03:02:19AM +0200, thejaguar@tutanota.de wrote:
So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ?
If I'm not mistaken the message will go to the destination's memory queue first; if the latter is full, only then will it make it to the diskq.
You can check the status of the queue using the control socket (syslog-ng-ctl stats)
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Fabien Wernli
-
Juhász, Viktor
-
Scheidler, Balázs
-
Szalai, Attila
-
thejaguar＠tutanota.de