<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>I'm running syslog-ng (with an elasticsearch2 destination configured) within a docker container, and I'm trying to avoid loss of messages if I kill the docker container and I start it again.<br></div><div><br></div><div>This is my scenary:</div><div><br></div><div>- A service which produces 20 lines of log per second<br></div><div>- A sislog-ng instance reading from a wildcard-file source (but actually it only reads logs from the above service, let's call it syslog-agent), which sends all logs to another syslog-ng instance (the one running in a docker container, let's call it syslog-relay) though a network destination.</div><div>- The syslog-relay sends messages to an elasticsearch instance, with following configuration:</div><div><br></div><div>options {<br>    chain-hostnames(no);<br>    use-dns(no);<br>    keep-hostname(yes);<br>    owner("syslog-ng");<br>    group("stratio");<br>    perm(0640);<br>    time-reap(30);<br>    mark-freq(10);<br>    stats-freq(0);<br>    bad-hostname("^gconfd$");<br>    flush-lines(100);<br>    log-fifo-size(1000);<br>    };<br><br></div><div><i>destination d_elastic_default_0 {<br>    elasticsearch2(<br>        cluster("myelastic")<br>        cluster-url("<a href="https://myelastic.logs:9200">https://myelastic.logs:9200</a>")<br>        client_mode("https")<br>        index("default")<br>        type("log")<br>        flush-limit(1)<br>        disk-buffer(<br>            mem-buf-size(16M)<br>            disk-buf-size(16M)<br>            reliable(yes)<br>            dir("/syslog-ng/log")<br>        )<br>        http-auth-type("clientcert")<br>        java-keystore-filepath("/etc/syslog-ng/certificates/syslog-relay.jks")<br>        java-keystore-password("XXXXXX")<br>        java-truststore-filepath("/etc/syslog-ng/certificates/ca-bundle.jks")<br>        java-truststore-password("XXXXXXXXXX")<br>    );<br>};</i><br><br></div><div>- The dir "/syslog-ng/log" is mapped to a path "/tmp/buffer" from the host where the docker container is running, so when I kill the docker container, the buffer file is not lost.</div><div>- I've set flush-limit to 1 because I thought that I may lost 1 message only as much.</div><div><br></div><div>This architecture is working fine (flush-limit=1 makes very slow, but for this test is ok), but if I kill the syslog-relay docker container, wait 5 to 10 seconds and start it again from scratch, I can see that several hundreds of logs are missing in elasticsearch. I check it by stopping the logger service and letting syslog-ng agent & relay to finish the process enqueued messages.<br></div><div><br></div><div>I can see in the syslog-agent stats that all logs messages have been processed, so it seems the problem is on the syslog-relay.  </div><div><br></div><div>Is this behaviour expected? If so, how can I protect against loss of messages in case of a syslog-relay docker container unexpected kill?</div><div><br></div><div>Thanks in advance.<br></div><div><br></div><div><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><span style="font-size:9pt;font-family:Tahoma;color:rgb(217,217,217);font-weight:700;vertical-align:baseline;white-space:pre-wrap">|</span><span style="font-size:9pt;font-family:Tahoma;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap"> Jose Angel Santiago</span><br></div><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.7273px"><span><br><p dir="ltr" style="line-height:1.656;margin-top:0pt;margin-bottom:0pt"><a href="http://www.stratio.com/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/SfSwo77PKD8TgM6em8B7mlcqOV9_N-AoAASRBMbZ3PFYgdFIibiMdr3zR_AZbMScWJOeRN7me-R_nK6vn1rnHSbApJVGfEjendjUa7LiGewC_fPGilVYciUS7E9v4mpKpl--caud" style="border: medium none;" alt="Logo_signature2.png" width="96" height="22"></span></a></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Vía de las dos Castillas, 33, Ática 4, 3ª Planta</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">28224 Pozuelo de Alarcón, Madrid, Spain </span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);vertical-align:baseline;white-space:pre-wrap">+34 918 286 473 | </span><a href="http://www.stratio.com" target="_blank"><span style="font-size:8pt;font-family:Tahoma;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap">www.stratio.com</span></a></p><a href="https://twitter.com/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/hOlHqJK94rZ7nBo9gzYKhLiwgogX1sgXV81pPDpAHHyRVeCjHxw0THNCq19zhcZalZiYeVYt9r4T_7LhoeLMxN1eTMnG46IfttV83WkTGC3jL1z04craZ8mmUn9hNnxDTIgh4_cT" style="border: medium none;" width="20" height="20"></span></a><a href="https://www.linkedin.com/company/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/xs2vZh2SrmNnOaJO9i07vQzFNoghAaZytG_Zh09D9-ESjgOv5LHzLrbVNOFa-e3g5FYdmeg-kj6Ur6hID_h1joaSCdsHETfxTNqNSXan5nxBdGtmxq6NMWoh6puraVK1JGPxzhzX" style="border: medium none;" width="20" height="20"></span></a><a href="https://www.youtube.com/c/StratioBD" target="_blank"><span style="font-size:8pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/GdwY5Qtr3qEaHfNZszPc0lGx52_bfO00F-ge1MzYQPNLQmXf2jO4z2kvWwIEKi2swQ-IfLjNvpS7o4wR0nwNhnhOFd7zZ1zJDtFVZLkp52XyaM0GpEXySD2iLbWq-dw0cmXUe7Jj" style="border: medium none;" width="20" height="20"></span></a></span><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div></div></div>