[syslog-ng] disk based buffering

[thejaguar at tutanota.de]

So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated.  The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host.  Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ?
Thanks


27. Sep 2016 12:41 by thejaguar at tutanota.de:


>     
> Thanks.> So if the queue stays intact, syslog-ng will try to send unsent messages as and when it starts ? even after 2-3 days ? it does not reset the queue or tracking  ever ?
> Thanks again
>
> 27. Sep 2016 12:11 by > balazs.scheidler at balabit.com> :
>
>
>>
>> Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact.
>>
>> There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package.
>>
>> On Sep 27, 2016 8:35 PM,  <>> thejaguar at tutanota.de>> > wrote:
>>
>>>           >>> Hi,>>> I have been using disk based buffering with reliable turned on yes as suggested here :->>> https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-diskbuffer-reliable.html
>>>
>>> This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now  syslog-ng is brilliant and sends all the stored/queued  logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?>>> I noticed any logs generated  in between power resets and which are not sent are not transmitted  on next net connection.  Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in  /var/lib/syslog-ng/syslog-ng-00000.rqf  or syslog-ng.persist ?
>>> =======================
>>>
>>> destination d_net {>>>         network (>>>                 "`myloghost`" port(`mylogport`) transport("tls")>>>                 tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) )>>>                 disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) )>>>                 template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n")>>>         );>>> };
>>> syslog-ng 3.8.1>>> Installer-Version: 3.8.1>>> Revision:>>> Module-Directory: /usr/lib/syslog-ng>>> Module-Path: /usr/lib/syslog-ng>>> Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsot>>> Enable-Debug: off>>> Enable-GProf: off>>> Enable-Memtrace: off>>> Enable-IPv6: off>>> Enable-Spoof-Source: off>>> Enable-TCP-Wrapper: off>>> Enable-Linux-Caps: off
>>> =======================
>>>
>>> Thanks
>>>
>>>   
>>> ______________________________________________________________________________
>>> Member info: >>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: >>> http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160928/8235d199/attachment-0001.htm