[syslog-ng] Disk file truncation issue
Peter Kokai (pkokai)
Peter.Kokai at oneidentity.com
Sun Dec 20 20:42:16 UTC 2020
Hello,
The linked repository has the latest version: 3.30.1
I would recommend the update as even since 3.13.3 there were many disk
queue related fixes.
The mentioned dqtool debug message was added to the 3.17.2 version.
Please share the output of dqtool that I shared in my previose e-mail.
--
Kokan
On Sat, Dec 19, 2020 at 11:17:21PM +0530, Raghunath Adhyapak wrote:
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
>
> Hi,
>
> # cat /etc/apt/sources.list.d/syslog-ng-obs.list
> deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04 ./
>
> # apt list | grep syslog-ng
>
> WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
>
> syslog-ng/bionic 3.13.2-3 all
> syslog-ng-core/bionic,now 3.13.2-3 amd64 [installed]
> syslog-ng-dbg/bionic 3.13.2-3 amd64
> syslog-ng-dev/bionic 3.13.2-3 amd64
> syslog-ng-mod-add-contextual-data/bionic 3.13.2-3 amd64
> syslog-ng-mod-amqp/bionic 3.13.2-3 amd64
> syslog-ng-mod-basicfuncs-plus/bionic 0.6.2-0.1 amd64
> syslog-ng-mod-extra/bionic 3.13.2-3 all
> syslog-ng-mod-geoip/bionic 3.13.2-3 amd64
> syslog-ng-mod-getent/bionic 3.13.2-3 amd64
> syslog-ng-mod-graphite/bionic 3.13.2-3 amd64
> syslog-ng-mod-grok/bionic 0.6.2-0.1 amd64
> syslog-ng-mod-journal/bionic,now 3.13.2-3 amd64 [installed,automatic]
> syslog-ng-mod-json/bionic 3.13.2-3 amd64
> syslog-ng-mod-kafka/bionic 0.6.2-0.1 amd64
> syslog-ng-mod-lua/bionic 0.6.2-0.1 amd64
> syslog-ng-mod-map-value-pairs/bionic 3.13.2-3 amd64
> syslog-ng-mod-mongodb/bionic 3.13.2-3 amd64
> syslog-ng-mod-perl/bionic 0.6.2-0.1 amd64
> syslog-ng-mod-python/bionic 3.13.2-3 amd64
> syslog-ng-mod-redis/bionic 3.13.2-3 amd64
> syslog-ng-mod-riemann/bionic 3.13.2-3 amd64
> syslog-ng-mod-rss/bionic 0.6.2-0.1 amd64
> syslog-ng-mod-smtp/bionic 3.13.2-3 amd64
> syslog-ng-mod-snmptrapd-parser/bionic 3.13.2-3 amd64
> syslog-ng-mod-sql/bionic 3.13.2-3 amd64
> syslog-ng-mod-stardate/bionic 3.13.2-3 amd64
> syslog-ng-mod-stomp/bionic 3.13.2-3 amd64
> syslog-ng-mod-tag-parser/bionic 3.13.2-3 amd64
> syslog-ng-mod-trigger/bionic 0.6.2-0.1 amd64
> syslog-ng-mod-xml-parser/bionic 3.13.2-3 amd64
> syslog-ng-mod-zmq/bionic 0.6.2-0.1 amd64
>
> Should it need an upgrade, please let me know.
>
> Thanks
> Raghu
>
> On Sat, Dec 19, 2020 at 11:00 PM Peter Kokai (pkokai) <Peter.Kokai at oneidentity.com<mailto:Peter.Kokai at oneidentity.com>> wrote:
> Hello,
>
> Issues like that have been fixed since 3.3 (you might want to check the latest version).
>
> If you have newer version of the dqtool (you can use that tool to
> investigate disk queue files without starting syslog-ng), there is an
> option to list internal states with the command
>
> dqtool info --debug --verbose <diskqueue-file>
>
> That should print out the following line:
>
> Reliable disk-buffer internal state; filename="...", backlog_head=X, read_head=Y, write_head=Z, backlog_len=W
>
> The backlog_-, read_- and wirte_head should be equal in case of empty
> disk queue files, otherwise there are still messages stored.
>
>
> --
> Kokan
>
> On Sat, Dec 19, 2020 at 10:29:35PM +0530, Raghunath Adhyapak wrote:
> > CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
> >
> > Hi,
> >
> > We have configured the destination with reliable(yes).
> > We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated.
> >
> > This syslog-ng service is running on 18.04 ubuntu VM.
> >
> > ===============================================
> > Our destination configuration is as below:
> > destination d_msg_syslog_01a_06 {
> > tcp(
> > "172.XX.XX.XX"
> > port(514)
> > # log-fifo-size should be atleast max-connections * log-fetch-limit
> > # 500 * 10 (default value of log-fetch-limit)
> > log-fifo-size(50000)
> > # throttle to max 5k eps
> > throttle(3000)
> > disk-buffer(
> > # number of bytes to store in memory
> > mem-buf-size(10000)
> > # number of bytes to store on disk
> > disk-buf-size(75161927680) # 70GB
> > reliable(yes)
> > # directory location to persist messages
> > dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/")
> > )
> > persist-name(d_msg_syslog_01a_06)
> > );
> > };
> > ===============================================
> > $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh
> > -rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf
> > -rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf
> >
> > Here are log snippets when we ran syslog-ng in debug mode.
> > ===============================================
> > [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat'
> > [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846'
> > [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0'
> > [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0'
> > [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0'
> > [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0'
> > [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> >
> > ===============================================
> >
> > We are unable to figure out why the disk buffer file is not getting truncated.
> > We have restarted syslog-ng multiple times.
> > Please advise.
> >
> > Thanks
> > Raghu
>
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
More information about the syslog-ng
mailing list