<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my special use case.<br></div><div><br></div><div>Unfortunately my syslog-ng agent reads some binary files which are piped through the log file I'm processing and sending it to elasticsearch. When this happens, the elastic bulk post fail (So I'am affected by this issue <a href="https://github.com/balabit/syslog-ng/issues/1993" target="_blank">https://github.com/balabit/syslog-ng/issues/1993</a>). I've debugged it and the problem was that message generated when applying the template is empty.</div><div><br></div><div>This is my network destination:<br></div><div><br></div><div>destination d_syslog_tcp {<br>    network(<br></div><div>        "syslog-relay.node"<br></div><div>        port(7601)<br>        template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n")<br>        template-escape(no)<br>        disk-buffer(<br>            mem-buf-size(200M)<br>            disk-buf-size(500M)<br>            reliable(yes)<br>            dir("/var/syslog-ng")<br>        )<br>    );<br>};<br></div><div><br></div>I've opened a netcat server as syslog-relay.node and piped a small binary file through the stdout file which I'm using as source, and this what the relay receives:</div><div dir="ltr"><br></div><div dir="ltr"><font size="1">2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 � <br>2018-12-17T15:35:57+00:00 agent-4 192.168.54.17  3F<br>2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 <br>                                                                                                                                        $<br>2018-12-17T15:35:57+00:00 agent-4 192.168.54.17  4) </font><br><br></div><div>As you can see, fourth message has no timestamp, hostname or ip address, it only contains "$"char, so when this message is sent to elastic, I've got a mapperException since it tries to validate the received timestamp.<br></div><div>I know I can customize elastic so it doesn't fail in case a wrong timestamp is received, but the point is that syslog-ng seems to fail when templating the message, doesn't it?</div><div><br></div><div>Also it would be great if syslog-ng had any mechanism to filter all those binary messages, I've already tried with filter expression, excluding all non-printable chars, but the issue still happens.</div><div><br></div><div>Regards.<br></div><div><br></div><div dir="ltr"><br></div><div dir="ltr"><div>-- <br><div dir="ltr" class="gmail-m_2269236669992505383gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><span style="font-size:9pt;font-family:Tahoma;color:rgb(217,217,217);font-weight:700;vertical-align:baseline;white-space:pre-wrap">|</span><span style="font-size:9pt;font-family:Tahoma;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap"> Jose Angel Santiago</span><br></div><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.7273px"><span><br><p dir="ltr" style="line-height:1.656;margin-top:0pt;margin-bottom:0pt"><a href="http://www.stratio.com/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/SfSwo77PKD8TgM6em8B7mlcqOV9_N-AoAASRBMbZ3PFYgdFIibiMdr3zR_AZbMScWJOeRN7me-R_nK6vn1rnHSbApJVGfEjendjUa7LiGewC_fPGilVYciUS7E9v4mpKpl--caud" style="border: medium none;" alt="Logo_signature2.png" width="96" height="22"></span></a></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Vía de las dos Castillas, 33, Ática 4, 3ª Planta</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">28224 Pozuelo de Alarcón, Madrid, Spain </span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);vertical-align:baseline;white-space:pre-wrap">+34 918 286 473 | </span><a href="http://www.stratio.com" target="_blank"><span style="font-size:8pt;font-family:Tahoma;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap">www.stratio.com</span></a></p><a href="https://twitter.com/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/hOlHqJK94rZ7nBo9gzYKhLiwgogX1sgXV81pPDpAHHyRVeCjHxw0THNCq19zhcZalZiYeVYt9r4T_7LhoeLMxN1eTMnG46IfttV83WkTGC3jL1z04craZ8mmUn9hNnxDTIgh4_cT" style="border: medium none;" width="20" height="20"></span></a><a href="https://www.linkedin.com/company/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/xs2vZh2SrmNnOaJO9i07vQzFNoghAaZytG_Zh09D9-ESjgOv5LHzLrbVNOFa-e3g5FYdmeg-kj6Ur6hID_h1joaSCdsHETfxTNqNSXan5nxBdGtmxq6NMWoh6puraVK1JGPxzhzX" style="border: medium none;" width="20" height="20"></span></a><a href="https://www.youtube.com/c/StratioBD" target="_blank"><span style="font-size:8pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/GdwY5Qtr3qEaHfNZszPc0lGx52_bfO00F-ge1MzYQPNLQmXf2jO4z2kvWwIEKi2swQ-IfLjNvpS7o4wR0nwNhnhOFd7zZ1zJDtFVZLkp52XyaM0GpEXySD2iLbWq-dw0cmXUe7Jj" style="border: medium none;" width="20" height="20"></span></a></span><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>