Re: [syslog-ng] disk based buffering
Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact. There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package. On Sep 27, 2016 8:35 PM, <thejaguar@tutanota.de> wrote: Hi, I have been using disk based buffering with reliable turned on yes as suggested here :- https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng- ose-guide-admin/html/configuring-diskbuffer-reliable.html This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ? I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? ======================= destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") ); }; syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite, pseudofile,afuser,kvformat,add-contextual-data,date, csvparser,linux-kmsg-format,confgen,syslogformat,afprog, disk-buffer,dbparser,afsot Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off ======================= Thanks ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks.So if the queue stays intact, syslog-ng will try to send unsent messages as and when it starts ? even after 2-3 days ? it does not reset the queue or tracking ever ? Thanks again 27. Sep 2016 12:11 by balazs.scheidler@balabit.com:
Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact.
There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package.
On Sep 27, 2016 8:35 PM, <> thejaguar@tutanota.de> > wrote:
>> Hi,>> I have been using disk based buffering with reliable turned on yes as suggested here :->> https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o...
This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?>> I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? =======================
destination d_net {>> network (>> "`myloghost`" port(`mylogport`) transport("tls")>> tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) )>> disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) )>> template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n")>> );>> }; syslog-ng 3.8.1>> Installer-Version: 3.8.1>> Revision:>> Module-Directory: /usr/lib/syslog-ng>> Module-Path: /usr/lib/syslog-ng>> Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsot>> Enable-Debug: off>> Enable-GProf: off>> Enable-Memtrace: off>> Enable-IPv6: off>> Enable-Spoof-Source: off>> Enable-TCP-Wrapper: off>> Enable-Linux-Caps: off =======================
Thanks
______________________________________________________________________________ Member info: >> https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: >> http://www.balabit.com/wiki/syslog-ng-faq
No, it resends everything it finds in the queue. On Sep 27, 2016 8:41 PM, <thejaguar@tutanota.de> wrote:
Thanks. So if the queue stays intact, syslog-ng will try to send unsent messages as and when it starts ? even after 2-3 days ? it does not reset the queue or tracking ever ?
Thanks again
27. Sep 2016 12:11 by balazs.scheidler@balabit.com:
Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact.
There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package.
On Sep 27, 2016 8:35 PM, <thejaguar@tutanota.de> wrote:
Hi, I have been using disk based buffering with reliable turned on yes as suggested here :- https://www.balabit.com/documents/syslog-ng-ose- latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-diskbuffer- reliable.html
This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ? I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ?
=======================
destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") ); };
syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Available-Modules: cef,affile,basicfuncs,system- source,cryptofuncs,graphite,pseudofile,afuser,kvformat, add-contextual-data,date,csvparser,linux-kmsg-format, confgen,syslogformat,afprog,disk-buffer,dbparser,afsot Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off
=======================
Thanks
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Scheidler, Balázs
-
thejaguar＠tutanota.de