[syslog-ng] Disk file truncation issue

Sat Dec 19 17:30:25 UTC 2020

Hello,

Issues like that have been fixed since 3.3 (you might want to check the latest version).

If you have newer version of the dqtool (you can use that tool to
investigate disk queue files without starting syslog-ng), there is an
option to list internal states with the command

dqtool info --debug --verbose <diskqueue-file>

That should print out the following line:

Reliable disk-buffer internal state; filename="...", backlog_head=X, read_head=Y, write_head=Z, backlog_len=W

The backlog_-, read_- and wirte_head should be equal in case of empty
disk queue files, otherwise there are still messages stored.

--
Kokan

On Sat, Dec 19, 2020 at 10:29:35PM +0530, Raghunath Adhyapak wrote:
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
> 
> Hi,
> 
> We have configured the destination with reliable(yes).
> We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated.
> 
> This syslog-ng service is running on 18.04 ubuntu VM.
> 
> ===============================================
> Our destination configuration is as below:
> destination d_msg_syslog_01a_06 {
>     tcp(
>         "172.XX.XX.XX"
>         port(514)
>         # log-fifo-size should be atleast max-connections * log-fetch-limit
>         # 500 * 10 (default value of log-fetch-limit)
>         log-fifo-size(50000)
>         # throttle to max 5k eps
>         throttle(3000)
>         disk-buffer(
>             # number of bytes to store in memory
>             mem-buf-size(10000)
>             # number of bytes to store on disk
>             disk-buf-size(75161927680) # 70GB
>             reliable(yes)
>             # directory location to persist messages
>             dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/")
>         )
>         persist-name(d_msg_syslog_01a_06)
>     );
> };
> ===============================================
> $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh
> -rw------- 1 root root  64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf
> -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf
> -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf
> -rw------- 1 root root  37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf
> -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf
> -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf
> 
> Here are log snippets when we ran syslog-ng in debug mode.
> ===============================================
> [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat'
> [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846'
> [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0'
> [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0'
> [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0'
> [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0'
> [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> 
> ===============================================
> 
> We are unable to figure out why the disk buffer file is not getting truncated.
> We have restarted syslog-ng multiple times.
> Please advise.
> 
> Thanks
> Raghu

> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 