[syslog-ng] Disk file truncation issue
Raghunath Adhyapak
funduraghu at gmail.com
Sat Dec 19 17:47:21 UTC 2020
Hi,
# cat /etc/apt/sources.list.d/syslog-ng-obs.list
deb
http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04
./
# apt list | grep syslog-ng
WARNING: apt does not have a stable CLI interface. Use with caution in
scripts.
syslog-ng/bionic 3.13.2-3 all
syslog-ng-core/bionic,now 3.13.2-3 amd64 [installed]
syslog-ng-dbg/bionic 3.13.2-3 amd64
syslog-ng-dev/bionic 3.13.2-3 amd64
syslog-ng-mod-add-contextual-data/bionic 3.13.2-3 amd64
syslog-ng-mod-amqp/bionic 3.13.2-3 amd64
syslog-ng-mod-basicfuncs-plus/bionic 0.6.2-0.1 amd64
syslog-ng-mod-extra/bionic 3.13.2-3 all
syslog-ng-mod-geoip/bionic 3.13.2-3 amd64
syslog-ng-mod-getent/bionic 3.13.2-3 amd64
syslog-ng-mod-graphite/bionic 3.13.2-3 amd64
syslog-ng-mod-grok/bionic 0.6.2-0.1 amd64
syslog-ng-mod-journal/bionic,now 3.13.2-3 amd64 [installed,automatic]
syslog-ng-mod-json/bionic 3.13.2-3 amd64
syslog-ng-mod-kafka/bionic 0.6.2-0.1 amd64
syslog-ng-mod-lua/bionic 0.6.2-0.1 amd64
syslog-ng-mod-map-value-pairs/bionic 3.13.2-3 amd64
syslog-ng-mod-mongodb/bionic 3.13.2-3 amd64
syslog-ng-mod-perl/bionic 0.6.2-0.1 amd64
syslog-ng-mod-python/bionic 3.13.2-3 amd64
syslog-ng-mod-redis/bionic 3.13.2-3 amd64
syslog-ng-mod-riemann/bionic 3.13.2-3 amd64
syslog-ng-mod-rss/bionic 0.6.2-0.1 amd64
syslog-ng-mod-smtp/bionic 3.13.2-3 amd64
syslog-ng-mod-snmptrapd-parser/bionic 3.13.2-3 amd64
syslog-ng-mod-sql/bionic 3.13.2-3 amd64
syslog-ng-mod-stardate/bionic 3.13.2-3 amd64
syslog-ng-mod-stomp/bionic 3.13.2-3 amd64
syslog-ng-mod-tag-parser/bionic 3.13.2-3 amd64
syslog-ng-mod-trigger/bionic 0.6.2-0.1 amd64
syslog-ng-mod-xml-parser/bionic 3.13.2-3 amd64
syslog-ng-mod-zmq/bionic 0.6.2-0.1 amd64
Should it need an upgrade, please let me know.
Thanks
Raghu
On Sat, Dec 19, 2020 at 11:00 PM Peter Kokai (pkokai) <
Peter.Kokai at oneidentity.com> wrote:
> Hello,
>
> Issues like that have been fixed since 3.3 (you might want to check the
> latest version).
>
> If you have newer version of the dqtool (you can use that tool to
> investigate disk queue files without starting syslog-ng), there is an
> option to list internal states with the command
>
> dqtool info --debug --verbose <diskqueue-file>
>
> That should print out the following line:
>
> Reliable disk-buffer internal state; filename="...", backlog_head=X,
> read_head=Y, write_head=Z, backlog_len=W
>
> The backlog_-, read_- and wirte_head should be equal in case of empty
> disk queue files, otherwise there are still messages stored.
>
>
> --
> Kokan
>
> On Sat, Dec 19, 2020 at 10:29:35PM +0530, Raghunath Adhyapak wrote:
> > CAUTION: This email originated from outside of the organization. Do not
> follow guidance, click links, or open attachments unless you recognize the
> sender and know the content is safe.
> >
> > Hi,
> >
> > We have configured the destination with reliable(yes).
> > We observe that many times the disk buffer file has zero messages,
> however the buffer file is not getting truncated.
> >
> > This syslog-ng service is running on 18.04 ubuntu VM.
> >
> > ===============================================
> > Our destination configuration is as below:
> > destination d_msg_syslog_01a_06 {
> > tcp(
> > "172.XX.XX.XX"
> > port(514)
> > # log-fifo-size should be atleast max-connections *
> log-fetch-limit
> > # 500 * 10 (default value of log-fetch-limit)
> > log-fifo-size(50000)
> > # throttle to max 5k eps
> > throttle(3000)
> > disk-buffer(
> > # number of bytes to store in memory
> > mem-buf-size(10000)
> > # number of bytes to store on disk
> > disk-buf-size(75161927680) # 70GB
> > reliable(yes)
> > # directory location to persist messages
> > dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/")
> > )
> > persist-name(d_msg_syslog_01a_06)
> > );
> > };
> > ===============================================
> > $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls
> -lh
> > -rw------- 1 root root 64G Dec 19 16:50
> /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 12:38
> /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 12:38
> /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf
> > -rw------- 1 root root 37G Dec 19 12:41
> /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 12:38
> /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf
> > -rw------- 1 root root 4.0K Dec 19 16:50
> /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf
> >
> > Here are log snippets when we ran syslog-ng in debug mode.
> > ===============================================
> > [2020-12-19T16:44:30.986904] Module loaded and initialized successfully;
> module='syslogformat'
> > [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded;
> filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf',
> queue_length='39418538', size='-1990797846'
> > [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were
> changed in syslog-ng 3.3, the configuration value was divided by the value
> of max-connections(). The result was too small, clamping to 100 entries.
> Ensure you have a proper log_fifo_size setting to avoid message loss.;
> orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded;
> filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf',
> queue_length='0', size='0'
> > [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were
> changed in syslog-ng 3.3, the configuration value was divided by the value
> of max-connections(). The result was too small, clamping to 100 entries.
> Ensure you have a proper log_fifo_size setting to avoid message loss.;
> orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded;
> filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf',
> queue_length='0', size='0'
> > [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were
> changed in syslog-ng 3.3, the configuration value was divided by the value
> of max-connections(). The result was too small, clamping to 100 entries.
> Ensure you have a proper log_fifo_size setting to avoid message loss.;
> orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded;
> filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf',
> queue_length='0', size='0'
> > [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were
> changed in syslog-ng 3.3, the configuration value was divided by the value
> of max-connections(). The result was too small, clamping to 100 entries.
> Ensure you have a proper log_fifo_size setting to avoid message loss.;
> orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> > [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded;
> filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf',
> queue_length='0', size='0'
> > [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were
> changed in syslog-ng 3.3, the configuration value was divided by the value
> of max-connections(). The result was too small, clamping to 100 entries.
> Ensure you have a proper log_fifo_size setting to avoid message loss.;
> orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'
> >
> > ===============================================
> >
> > We are unable to figure out why the disk buffer file is not getting
> truncated.
> > We have restarted syslog-ng multiple times.
> > Please advise.
> >
> > Thanks
> > Raghu
>
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201219/f5844b4f/attachment.html>
More information about the syslog-ng
mailing list