<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
That was it. I moved <span style="font-family: "Courier New", monospace; font-size: 10pt;">
syslog-ng.persist</span> to the persistent volume so it survives restarts and now everything works as expected.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you Kokan!</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Peter Kokai (pkokai) <Peter.Kokai@oneidentity.com><br>
<b>Sent:</b> Monday, March 22, 2021 12:31<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] Destination with disk-buffer and reliable(yes) looses queued messages on restart/crash?</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hello,<br>
<br>
Saving the queue file alone won't help you as you experienced that. There is also a persist file, the persist file contains a connection between a destination and its destination queue.<br>
<br>
A destination if diskq configured first checks the persist file, if there is an entry that says which diskq file it should use.<br>
If there is no entry in the persist file, it simply creates a new diskq file and files the entry.<br>
So if the persist file is not kept, syslog-ng just going to create a new diskq file.<br>
<br>
--<br>
kokan<br>
<br>
________________________________________<br>
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Ralf.Steppacher@swisscom.com <Ralf.Steppacher@swisscom.com><br>
Sent: 22 March 2021 12:13<br>
To: syslog-ng@lists.balabit.hu<br>
Subject: [syslog-ng] Destination with disk-buffer and reliable(yes) looses queued messages on restart/crash?<br>
<br>
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
<br>
Hello,<br>
<br>
I am probably missing something here: I am trying to configure syslog-ng (3.27.1) running in a Kubernetes Pod, using a Persistent Volume mounted at /var/log, such that queued messages are spooled to disk and in the event of a crash of syslog-ng the queue can
be recovered. I configured the destination like so:<br>
<br>
<br>
destination d_forwarder {<br>
syslog(<br>
"`HEAVY_FORWARDER_HOST`" port(`HEAVY_FORWARDER_PORT`)<br>
transport("tls")<br>
tls(<br>
ca-dir("/etc/syslog-ng/ca.d")<br>
key-file("/vault/secrets/client_key.pem")<br>
cert-file("/vault/secrets/client_cert.pem")<br>
peer-verify(required-trusted)<br>
)<br>
disk-buffer(<br>
mem-buf-size(524288)<br>
disk-buf-size(104857600)<br>
reliable(yes)<br>
dir("/var/log")<br>
)<br>
);<br>
};<br>
<br>
The documentation for the `reliable` flag says: "If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash."<br>
I can see several *.rqf files being created in /var/log. As soon as the latest of them reaches roughly 100MB messages start to get dropped. So far everything as expected.<br>
<br>
Stats:<br>
{<br>
"center_queued_processed": 72031,<br>
"center_received_processed": 36016,<br>
"destination_d_forwarder_processed": 36015,<br>
"destination_d_local_processed": 36016,<br>
"dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_dropped": 4173,<br>
"dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_processed": 36015,<br>
"dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_queued": 31842,<br>
"dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_written": 0,<br>
...<br>
"source_s_external_tls_processed": 36015<br>
...<br>
}<br>
<br>
/var/log:<br>
I have no name!@syslog-ng-76f898f5bb-sh9q8:/var/log$ ls -lh<br>
total 213M<br>
-rw------- 1 10001 10001 38K Mar 22 08:33 syslog-ng-00000.rqf<br>
-rw------- 1 10001 10001 4.0K Mar 22 09:25 syslog-ng-00001.rqf<br>
-rw------- 1 10001 10001 101M Mar 22 09:29 syslog-ng-00002.rqf<br>
-rw------- 1 10001 10001 4.0K Mar 22 09:46 syslog-ng-00003.rqf<br>
-rw------- 1 10001 10001 4.0K Mar 22 10:14 syslog-ng-00004.rqf<br>
...<br>
<br>
Now, if I kill the syslog-ng Pod or gracefully scale the deployment to 0 and back up to 1, the queue is still lost. The stats all go back to 0 and bringing up the destination shows no (queued) messages coming in. On every restart a new .rqf gets created. New
messages get spooled to the latest .rqf file until that one reaches the configured 100Mb size limit as well.<br>
<br>
What am I missing here?<br>
<br>
<br>
Thanks in advance!<br>
Ralf<br>
______________________________________________________________________________<br>
Member info: <a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CRalf.Steppacher%40swisscom.com%7C11d0342b856c419daefa08d8ed26099b%7C364e5b87c1c7420d9beec35d19b557a1%7C1%7C0%7C637520094907954300%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BWBv4mwueQ0Lc0lpPvLUXOO52IOEs9xNiAfktAuNXPw%3D&reserved=0">
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CRalf.Steppacher%40swisscom.com%7C11d0342b856c419daefa08d8ed26099b%7C364e5b87c1c7420d9beec35d19b557a1%7C1%7C0%7C637520094907954300%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BWBv4mwueQ0Lc0lpPvLUXOO52IOEs9xNiAfktAuNXPw%3D&reserved=0</a><br>
Documentation: <a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CRalf.Steppacher%40swisscom.com%7C11d0342b856c419daefa08d8ed26099b%7C364e5b87c1c7420d9beec35d19b557a1%7C1%7C0%7C637520094907954300%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Gu2hdej5hBh2Rq%2Bzz2iGA6MLQMGqJ9rSqYTqVZnNWEo%3D&reserved=0">
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CRalf.Steppacher%40swisscom.com%7C11d0342b856c419daefa08d8ed26099b%7C364e5b87c1c7420d9beec35d19b557a1%7C1%7C0%7C637520094907954300%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Gu2hdej5hBh2Rq%2Bzz2iGA6MLQMGqJ9rSqYTqVZnNWEo%3D&reserved=0</a><br>
FAQ: <a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CRalf.Steppacher%40swisscom.com%7C11d0342b856c419daefa08d8ed26099b%7C364e5b87c1c7420d9beec35d19b557a1%7C1%7C0%7C637520094907954300%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qYRrUXOJ3eL8m8IE1vFfPggohaV4Wq3rHFYWpBejGSo%3D&reserved=0">
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CRalf.Steppacher%40swisscom.com%7C11d0342b856c419daefa08d8ed26099b%7C364e5b87c1c7420d9beec35d19b557a1%7C1%7C0%7C637520094907954300%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qYRrUXOJ3eL8m8IE1vFfPggohaV4Wq3rHFYWpBejGSo%3D&reserved=0</a><br>
<br>
</div>
</span></font></div>
</body>
</html>