<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div># cat /etc/apt/sources.list.d/syslog-ng-obs.list</div><div>deb <a href="http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04">http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04</a> ./ </div><div><br></div><div><div><div># apt list | grep syslog-ng</div><div><br></div><div>WARNING: apt does not have a stable CLI interface. Use with caution in scripts.</div><div><br></div><div>syslog-ng/bionic 3.13.2-3 all</div><div><span style="background-color:rgb(255,255,0)">syslog-ng-core/bionic,now 3.13.2-3 amd64 [installed]</span></div><div>syslog-ng-dbg/bionic 3.13.2-3 amd64</div><div>syslog-ng-dev/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-add-contextual-data/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-amqp/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-basicfuncs-plus/bionic 0.6.2-0.1 amd64</div><div>syslog-ng-mod-extra/bionic 3.13.2-3 all</div><div>syslog-ng-mod-geoip/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-getent/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-graphite/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-grok/bionic 0.6.2-0.1 amd64</div><div>syslog-ng-mod-journal/bionic,now 3.13.2-3 amd64 [installed,automatic]</div><div>syslog-ng-mod-json/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-kafka/bionic 0.6.2-0.1 amd64</div><div>syslog-ng-mod-lua/bionic 0.6.2-0.1 amd64</div><div>syslog-ng-mod-map-value-pairs/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-mongodb/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-perl/bionic 0.6.2-0.1 amd64</div><div>syslog-ng-mod-python/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-redis/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-riemann/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-rss/bionic 0.6.2-0.1 amd64</div><div>syslog-ng-mod-smtp/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-snmptrapd-parser/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-sql/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-stardate/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-stomp/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-tag-parser/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-trigger/bionic 0.6.2-0.1 amd64</div><div>syslog-ng-mod-xml-parser/bionic 3.13.2-3 amd64</div><div>syslog-ng-mod-zmq/bionic 0.6.2-0.1 amd64</div></div></div><div><br></div><div>Should it need an upgrade, please let me know.</div><div><br></div><div>Thanks</div><div>Raghu</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Dec 19, 2020 at 11:00 PM Peter Kokai (pkokai) <<a href="mailto:Peter.Kokai@oneidentity.com">Peter.Kokai@oneidentity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
Issues like that have been fixed since 3.3 (you might want to check the latest version).<br>
<br>
If you have newer version of the dqtool (you can use that tool to<br>
investigate disk queue files without starting syslog-ng), there is an<br>
option to list internal states with the command<br>
<br>
dqtool info --debug --verbose <diskqueue-file><br>
<br>
That should print out the following line:<br>
<br>
Reliable disk-buffer internal state; filename="...", backlog_head=X, read_head=Y, write_head=Z, backlog_len=W<br>
<br>
The backlog_-, read_- and wirte_head should be equal in case of empty<br>
disk queue files, otherwise there are still messages stored.<br>
<br>
<br>
--<br>
Kokan<br>
<br>
On Sat, Dec 19, 2020 at 10:29:35PM +0530, Raghunath Adhyapak wrote:<br>
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
> <br>
> Hi,<br>
> <br>
> We have configured the destination with reliable(yes).<br>
> We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated.<br>
> <br>
> This syslog-ng service is running on 18.04 ubuntu VM.<br>
> <br>
> ===============================================<br>
> Our destination configuration is as below:<br>
> destination d_msg_syslog_01a_06 {<br>
> tcp(<br>
> "172.XX.XX.XX"<br>
> port(514)<br>
> # log-fifo-size should be atleast max-connections * log-fetch-limit<br>
> # 500 * 10 (default value of log-fetch-limit)<br>
> log-fifo-size(50000)<br>
> # throttle to max 5k eps<br>
> throttle(3000)<br>
> disk-buffer(<br>
> # number of bytes to store in memory<br>
> mem-buf-size(10000)<br>
> # number of bytes to store on disk<br>
> disk-buf-size(75161927680) # 70GB<br>
> reliable(yes)<br>
> # directory location to persist messages<br>
> dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/")<br>
> )<br>
> persist-name(d_msg_syslog_01a_06)<br>
> );<br>
> };<br>
> ===============================================<br>
> $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh<br>
> -rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf<br>
> -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf<br>
> -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf<br>
> -rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf<br>
> -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf<br>
> -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf<br>
> <br>
> Here are log snippets when we ran syslog-ng in debug mode.<br>
> ===============================================<br>
> [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat'<br>
> [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846'<br>
> [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'<br>
> [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0'<br>
> [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'<br>
> [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0'<br>
> [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'<br>
> [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0'<br>
> [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'<br>
> [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0'<br>
> [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000'<br>
> <br>
> ===============================================<br>
> <br>
> We are unable to figure out why the disk buffer file is not getting truncated.<br>
> We have restarted syslog-ng multiple times.<br>
> Please advise.<br>
> <br>
> Thanks<br>
> Raghu<br>
<br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
> <br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>