Hi,

We have configured the destination with reliable(yes).

We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated.

This syslog-ng service is running on 18.04 ubuntu VM.

===============================================

Our destination configuration is as below:

destination d_msg_syslog_01a_06 {

tcp(

"172.XX.XX.XX"

port(514)

# log-fifo-size should be atleast max-connections * log-fetch-limit

# 500 * 10 (default value of log-fetch-limit)

log-fifo-size(50000)

# throttle to max 5k eps

throttle(3000)

disk-buffer(

# number of bytes to store in memory

mem-buf-size(10000)

# number of bytes to store on disk

disk-buf-size(75161927680) # 70GB

reliable(yes)

# directory location to persist messages

dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/")

)

persist-name(d_msg_syslog_01a_06)

);

};

===============================================

$ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh

-rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf

-rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf

-rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf

-rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf

-rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf

-rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf

Here are log snippets when we ran syslog-ng in debug mode.

===============================================

[2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat'

[2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846'

[2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'

[2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0'

[2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'

[2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0'

[2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'

[2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0'

[2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'

[2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0'

[2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'

===============================================

We are unable to figure out why the disk buffer file is not getting truncated.

We have restarted syslog-ng multiple times.

Please advise.

Thanks

Raghu