<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am probably missing something here: I am trying to configure syslog-ng (3.27.1) running in a Kubernetes Pod, using a Persistent Volume mounted at
<span style="font-family: "Courier New", monospace;">/var/log</span>, such that queued messages are spooled to disk and in the event of a crash of syslog-ng the queue can be recovered. I configured the destination like so:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<pre style="background-color:#ffffff;color:#080808;font-family:'JetBrains Mono',monospace;font-size:9.8pt">destination d_forwarder {<br> syslog(<br> <span style="color:#067d17">"`HEAVY_FORWARDER_HOST`"</span> port(`HEAVY_FORWARDER_PORT`)<br> transport(<span style="color:#067d17">"tls"</span>)<br> tls(<br> ca-dir(<span style="color:#067d17">"/etc/syslog-ng/ca.d"</span>)<br> key-file(<span style="color:#067d17">"/vault/secrets/client_key.pem"</span>)<br> cert-file(<span style="color:#067d17">"/vault/secrets/client_cert.pem"</span>)<br> peer-verify(required-trusted)<br> )<br> disk-buffer(<br> mem-buf-size(524288)<br> disk-buf-size(104857600)<br> reliable(yes)<br> dir(<span style="color:#067d17">"/var/log"</span>)<br> )<br> )<span style="color:#8c8c8c;font-style:italic">;<br></span>}<span style="color:#8c8c8c;font-style:italic">;</span></pre>
The documentation for the `reliable` flag says: "<i>If set to <span class="Code">
yes</span>, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash.</i>"<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I can see several *.rqf files being created in /var/log. As soon as the latest of them reaches roughly 100MB messages start to get dropped. So far everything as expected.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Stats:<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace; font-size: 10pt;">{</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "center_queued_processed": 72031,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "center_received_processed": 36016,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "destination_d_forwarder_processed": 36015,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "destination_d_local_processed": 36016,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_dropped": 4173,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_processed": 36015,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_queued": 31842,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_written": 0,</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> ...<br>
</span></div>
<span style="font-family: "Courier New", monospace; font-size: 10pt;"> "source_s_external_tls_processed": 36015</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace; font-size: 10pt;"> ...<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace; font-size: 10pt;">}</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
/var/log:<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace; font-size: 10pt;">I have no name!@syslog-ng-76f898f5bb-sh9q8:/var/log$ ls -lh
</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">total 213M</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">-rw------- 1 10001 10001 38K Mar 22 08:33 syslog-ng-00000.rqf</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">-rw------- 1 10001 10001 4.0K Mar 22 09:25 syslog-ng-00001.rqf</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">-rw------- 1 10001 10001 101M Mar 22 09:29 syslog-ng-00002.rqf</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">-rw------- 1 10001 10001 4.0K Mar 22 09:46 syslog-ng-00003.rqf</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">-rw------- 1 10001 10001 4.0K Mar 22 10:14 syslog-ng-00004.rqf</span></div>
<span style="font-family: "Courier New", monospace; font-size: 10pt;">...</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Now, if I kill the syslog-ng Pod or gracefully scale the deployment to 0 and back up to 1, the queue is still lost. The stats all go back to 0 and bringing up the destination shows no (queued) messages coming in. On every restart a new .rqf gets created. New
messages get spooled to the latest .rqf file until that one reaches the configured 100Mb size limit as well.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
What am I missing here?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks in advance!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ralf<br>
</div>
</body>
</html>