[syslog-ng] Disk file truncation issue

[Raghunath Adhyapak]

Hi,

We have configured the destination with reliable(yes).
We observe that many times the disk buffer file has zero messages, however
the buffer file is not getting truncated.

This syslog-ng service is running on 18.04 ubuntu VM.

===============================================
Our destination configuration is as below:
destination d_msg_syslog_01a_06 {
    tcp(
        "172.XX.XX.XX"
        port(514)
        # log-fifo-size should be atleast max-connections * log-fetch-limit
        # 500 * 10 (default value of log-fetch-limit)
        log-fifo-size(50000)
        # throttle to max 5k eps
        throttle(3000)
        disk-buffer(
            # number of bytes to store in memory
            mem-buf-size(10000)
            # number of bytes to store on disk
            disk-buf-size(75161927680) # 70GB
            reliable(yes)
            # directory location to persist messages
            dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/")
        )
        persist-name(d_msg_syslog_01a_06)
    );
};
===============================================
$ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh
-rw------- 1 root root  64G Dec 19 16:50
/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf
-rw------- 1 root root 4.0K Dec 19 12:38
/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf
-rw------- 1 root root 4.0K Dec 19 12:38
/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf
-rw------- 1 root root  37G Dec 19 12:41
/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf
-rw------- 1 root root 4.0K Dec 19 12:38
/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf
-rw------- 1 root root 4.0K Dec 19 16:50
/datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf

Here are log snippets when we ran syslog-ng in debug mode.
===============================================
[2020-12-19T16:44:30.986904] Module loaded and initialized successfully;
module='syslogformat'
[2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded;
filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf',
queue_length='39418538', size='-1990797846'
[2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were
changed in syslog-ng 3.3, the configuration value was divided by the value
of max-connections(). The result was too small, clamping to 100 entries.
Ensure you have a proper log_fifo_size setting to avoid message loss.;
orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
[2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded;
filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf',
queue_length='0', size='0'
[2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were
changed in syslog-ng 3.3, the configuration value was divided by the value
of max-connections(). The result was too small, clamping to 100 entries.
Ensure you have a proper log_fifo_size setting to avoid message loss.;
orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
[2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded;
filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf',
queue_length='0', size='0'
[2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were
changed in syslog-ng 3.3, the configuration value was divided by the value
of max-connections(). The result was too small, clamping to 100 entries.
Ensure you have a proper log_fifo_size setting to avoid message loss.;
orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
[2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded;
filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf',
queue_length='0', size='0'
[2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were
changed in syslog-ng 3.3, the configuration value was divided by the value
of max-connections(). The result was too small, clamping to 100 entries.
Ensure you have a proper log_fifo_size setting to avoid message loss.;
orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
[2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded;
filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf',
queue_length='0', size='0'
[2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were
changed in syslog-ng 3.3, the configuration value was divided by the value
of max-connections(). The result was too small, clamping to 100 entries.
Ensure you have a proper log_fifo_size setting to avoid message loss.;
orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'

===============================================

We are unable to figure out why the disk buffer file is not getting
truncated.
We have restarted syslog-ng multiple times.
Please advise.

Thanks
Raghu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201219/bee3b03d/attachment.html>