<div dir="ltr"><div>Hi,</div><div><br></div><div>I guess I know what is happening, when I start from scratch the docker container, even I provide a persist file and a buffer file within the mapped volume, syslog-ng recreates them so all messages in buffer file which were not processed by the relay are lost.</div><div><br></div><div>Is there any way to tell syslog-ng to use an already existing .persist file so it doesn't recreate the .rqf file?</div><div><br></div><div>Regards.<br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-09-13 16:23 GMT+02:00 Budai, László <span dir="ltr"><<a href="mailto:laszlo.budai@oneidentity.com" target="_blank">laszlo.budai@oneidentity.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>one problem could be if the flush-limit would be greater than 1... in that case syslog-ng would use a HttpBulkMessageProcessor.</div><div>In this case syslog-ng pass the message to the HttpBulkMessageProcessor and sends back a positive ACK to the LogSource (so the message is removed from the diskbuffer), and if the dockerimage is killed, all the messages stored in the HttpBulkMessageProcessor are lost.</div><div>But in your case syslog-ng should use the HttpSingleMessageProcessor... which means that the messages are sent one-by-one... <br></div><div>Could you check the diskbuffer with the dqtool?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br></div><div>L.<br></div></font></span><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Sep 13, 2018 at 3:50 PM, Jose Angel Santiago <span dir="ltr"><<a href="mailto:jasantiago@stratio.com" target="_blank">jasantiago@stratio.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>I'm running syslog-ng (with an elasticsearch2 destination configured) within a docker container, and I'm trying to avoid loss of messages if I kill the docker container and I start it again.<br></div><div><br></div><div>This is my scenary:</div><div><br></div><div>- A service which produces 20 lines of log per second<br></div><div>- A sislog-ng instance reading from a wildcard-file source (but actually it only reads logs from the above service, let's call it syslog-agent), which sends all logs to another syslog-ng instance (the one running in a docker container, let's call it syslog-relay) though a network destination.</div><div>- The syslog-relay sends messages to an elasticsearch instance, with following configuration:</div><div><br></div><div>options {<br> chain-hostnames(no);<br> use-dns(no);<br> keep-hostname(yes);<br> owner("syslog-ng");<br> group("stratio");<br> perm(0640);<br> time-reap(30);<br> mark-freq(10);<br> stats-freq(0);<br> bad-hostname("^gconfd$");<br> flush-lines(100);<br> log-fifo-size(1000);<br> };<br><br></div><div><i>destination d_elastic_default_0 {<br> elasticsearch2(<br> cluster("myelastic")<br> cluster-url("<a href="https://myelastic.logs:9200" target="_blank">https://myelastic<wbr>.logs:9200</a>")<br> client_mode("https")<br> index("default")<br> type("log")<br> flush-limit(1)<br> disk-buffer(<br> mem-buf-size(16M)<br> disk-buf-size(16M)<br> reliable(yes)<br> dir("/syslog-ng/log")<br> )<br> http-auth-type("clientcert")<br> java-keystore-filepath("/etc/s<wbr>yslog-ng/certificates/syslog-r<wbr>elay.jks")<br> java-keystore-password("XXXXXX<wbr>")<br> java-truststore-filepath("/etc<wbr>/syslog-ng/certificates/ca-bun<wbr>dle.jks")<br> java-truststore-password("XXXX<wbr>XXXXXX")<br> );<br>};</i><br><br></div><div>- The dir "/syslog-ng/log" is mapped to a path "/tmp/buffer" from the host where the docker container is running, so when I kill the docker container, the buffer file is not lost.</div><div>- I've set flush-limit to 1 because I thought that I may lost 1 message only as much.</div><div><br></div><div>This architecture is working fine (flush-limit=1 makes very slow, but for this test is ok), but if I kill the syslog-relay docker container, wait 5 to 10 seconds and start it again from scratch, I can see that several hundreds of logs are missing in elasticsearch. I check it by stopping the logger service and letting syslog-ng agent & relay to finish the process enqueued messages.<br></div><div><br></div><div>I can see in the syslog-agent stats that all logs messages have been processed, so it seems the problem is on the syslog-relay. </div><div><br></div><div>Is this behaviour expected? If so, how can I protect against loss of messages in case of a syslog-relay docker container unexpected kill?</div><div><br></div><div>Thanks in advance.<br></div><div><br></div><div><br>-- <br><div class="m_-7154903072736908858m_5742229891460430693m_-7066921092232463927gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><span style="font-size:9pt;font-family:Tahoma;color:rgb(217,217,217);font-weight:700;vertical-align:baseline;white-space:pre-wrap">|</span><span style="font-size:9pt;font-family:Tahoma;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap"> Jose Angel Santiago</span><br></div><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.7273px"><span><br><p dir="ltr" style="line-height:1.656;margin-top:0pt;margin-bottom:0pt"><a href="http://www.stratio.com/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/SfSwo77PKD8TgM6em8B7mlcqOV9_N-AoAASRBMbZ3PFYgdFIibiMdr3zR_AZbMScWJOeRN7me-R_nK6vn1rnHSbApJVGfEjendjUa7LiGewC_fPGilVYciUS7E9v4mpKpl--caud" style="border:medium none" alt="Logo_signature2.png" width="96" height="22"></span></a></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Vía de las dos Castillas, 33, Ática 4, 3ª Planta</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">28224 Pozuelo de Alarcón, Madrid, Spain </span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);vertical-align:baseline;white-space:pre-wrap">+34 918 286 473 | </span><a href="http://www.stratio.com" target="_blank"><span style="font-size:8pt;font-family:Tahoma;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap">www.stratio.com</span></a></p><a href="https://twitter.com/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/hOlHqJK94rZ7nBo9gzYKhLiwgogX1sgXV81pPDpAHHyRVeCjHxw0THNCq19zhcZalZiYeVYt9r4T_7LhoeLMxN1eTMnG46IfttV83WkTGC3jL1z04craZ8mmUn9hNnxDTIgh4_cT" style="border:medium none" width="20" height="20"></span></a><a href="https://www.linkedin.com/company/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/xs2vZh2SrmNnOaJO9i07vQzFNoghAaZytG_Zh09D9-ESjgOv5LHzLrbVNOFa-e3g5FYdmeg-kj6Ur6hID_h1joaSCdsHETfxTNqNSXan5nxBdGtmxq6NMWoh6puraVK1JGPxzhzX" style="border:medium none" width="20" height="20"></span></a><a href="https://www.youtube.com/c/StratioBD" target="_blank"><span style="font-size:8pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/GdwY5Qtr3qEaHfNZszPc0lGx52_bfO00F-ge1MzYQPNLQmXf2jO4z2kvWwIEKi2swQ-IfLjNvpS7o4wR0nwNhnhOFd7zZ1zJDtFVZLkp52XyaM0GpEXySD2iLbWq-dw0cmXUe7Jj" style="border:medium none" width="20" height="20"></span></a></span><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div></div></div>
<br></div></div><span class="">______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></span></blockquote></div><br></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div><div style="color:rgb(136,136,136);font-size:12.8000001907349px"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:12.7272720336914px"><span style="font-size:9pt;font-family:Tahoma;color:rgb(217,217,217);font-weight:700;vertical-align:baseline;white-space:pre-wrap">|</span><span style="font-size:9pt;font-family:Tahoma;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap"> Jose Angel Santiago</span><br></div><div style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:12.7272720336914px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.7272720336914px"><span><br><p dir="ltr" style="line-height:1.656;margin-top:0pt;margin-bottom:0pt"><a href="http://www.stratio.com/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/SfSwo77PKD8TgM6em8B7mlcqOV9_N-AoAASRBMbZ3PFYgdFIibiMdr3zR_AZbMScWJOeRN7me-R_nK6vn1rnHSbApJVGfEjendjUa7LiGewC_fPGilVYciUS7E9v4mpKpl--caud" width="96" height="22" style="border:none" alt="Logo_signature2.png"></span></a></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Vía de las dos Castillas, 33, Ática 4, 3ª Planta</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">28224 Pozuelo de Alarcón, Madrid, Spain </span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);vertical-align:baseline;white-space:pre-wrap">+34 918 286 473 | </span><a href="http://www.stratio.com" target="_blank"><span style="font-size:8pt;font-family:Tahoma;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap">www.stratio.com</span></a></p><a href="https://twitter.com/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/hOlHqJK94rZ7nBo9gzYKhLiwgogX1sgXV81pPDpAHHyRVeCjHxw0THNCq19zhcZalZiYeVYt9r4T_7LhoeLMxN1eTMnG46IfttV83WkTGC3jL1z04craZ8mmUn9hNnxDTIgh4_cT" width="20" height="20" style="border:none"></span></a><a href="https://www.linkedin.com/company/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/xs2vZh2SrmNnOaJO9i07vQzFNoghAaZytG_Zh09D9-ESjgOv5LHzLrbVNOFa-e3g5FYdmeg-kj6Ur6hID_h1joaSCdsHETfxTNqNSXan5nxBdGtmxq6NMWoh6puraVK1JGPxzhzX" width="20" height="20" style="border:none"></span></a><a href="https://www.youtube.com/c/StratioBD" target="_blank"><span style="font-size:8pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/GdwY5Qtr3qEaHfNZszPc0lGx52_bfO00F-ge1MzYQPNLQmXf2jO4z2kvWwIEKi2swQ-IfLjNvpS7o4wR0nwNhnhOFd7zZ1zJDtFVZLkp52XyaM0GpEXySD2iLbWq-dw0cmXUe7Jj" width="20" height="20" style="border:none"></span></a></span><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>