<div dir="ltr"><div>Using the indent-multi-line along with sanitize function, I've managed to send every message perfectly parsed to elasticsearch.</div><div><br></div><div>Thank you so much for your excellent support!!<br></div></div><br><div class="gmail_quote"><div dir="ltr">El jue., 20 dic. 2018 a las 13:59, Scheidler, Balázs (<<a href="mailto:balazs.scheidler@oneidentity.com">balazs.scheidler@oneidentity.com</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>If I understand everything right, then the problem is that you are reading the output as a multi-line capable file source multi-line-mode(indented).</div><div><br></div><div>This means that your input file will contain messages that span multiple lines. When you read that into syslog-ng, your $MSG macro will have an embedded newline character in it.</div><div><br></div><div>With that said, you are using this template: <i>template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA} ${MESSAGE}\n")</i></div><div><i><br></i></div><div>So this could produce a header (with ISODATE and everything else), and at the end the $MESSAGE value, with NL characters in it.</div><div><br></div><div>So whenever you see an empty line, that's because of the value of $MESSAGE.</div><div><br></div><div>To confirm that this is the case, please enclose the ${MESSAGE} macro in some visible characters, like this:</div><div><br></div><div><i>template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA} >>>${MESSAGE}<<<\n")</i></div><div><i><br></i></div><div>You will see that the newline will be within >>> and <<<.</div><div><br></div><div>To resolve the situation please consider:</div><div>1) whether you actually need a multi-line capable source. By the look of it, you are reading stdout files, line-by-line. Working in multi-line mode means that a single message will span multiple input lines.</div><div>2) if you do (for instance to capture a multi-line backtrace as a single message), you can always use "$(indent-multi-line $MESSAGE)" in place of ${MESSAGE} in the template above. That will insert whitespace into the output file at NL barriers, so that records can still be reconstructed.</div><div>3) if you selectively need multi-line messages in one destination and not in another, you can always use flags(no-multi-line) at the destination, which will simply remove NL characters _after_ the template has been formatted.</div><div><br></div><div>Also, you will find, that the template() mechanism is not capable of working with embedded NUL characters, so if you happen to capture an embedded NUL into a message, due to the multi-line mechanism, that can terminate the template() based output somewhat earlier.</div><div><br></div><div>You can also remove invalid characters using the $(sanitize) template function, e.g. $(sanitize $MESSAGE). $(sanitize) has the following syntax:</div><div><br></div><div>/*<br> * $(sanitize [opts] $arg1 $arg2 ...)<br> *<br> * Options:<br> * --ctrl-chars or -c Filter control characters (default)<br> * --no-ctrl-chars or -C Don't filter control characters<br> * --invalid-chars <set> or -i Set of characters to be translated, default "/"<br> * --replacement <replace> or -r Single character replacement for invalid chars.<br> */<br></div><div>This removes all control characters, and the set specified in --invalid-chars, which defaults to '/'.<br></div><div><br></div><div><i></i></div><div><i></i></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Dec 20, 2018 at 11:11 AM Jose Angel Santiago <<a href="mailto:jasantiago@stratio.com" target="_blank">jasantiago@stratio.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div>I redirected my netcat listener output to a file, and I don't see any blank lines on that file. So I decided to put a netcat as the final destination (i.e syslog-agent -> syslog-relay -> netcat) redirecting its output to a file. <br></div><div><br></div><div>I've attached a sample of the the text plain file, where you can see a blank message (with would make elasticsearck bulk fail)</div><div>I've also attached the hexdump of that file.</div><div><br></div><div> </div><div>I'm using the dqtool file as a binary file for my tests.</div><div><br></div><div>Here is my source statement for syslog-agent:<br></div><div><br></div><div><i>source s_stdout_files { wildcard-file(<br> base-dir("/logs")<br> filename-pattern("stdout")<br> max-files(100000)<br> follow-freq(0.5)<br> monitor-method(poll)<br> log-fetch-limit(100)<br> log-iw-size(15000)<br> flags(no-parse)<br> recursive(yes)<br> multi-line-mode(indented)<br>); };</i></div><div><br></div><div>Here is my source statement for syslog-relay:</div><div><br></div><div><font size="2"><i>source s_network {<br> network(<br> port(7601)<br> transport("tls")<br> flags(no-multi-line, no-parse)<br> tls( key-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.key")<br> cert-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.pem")<br> ca-dir("/etc/syslog-ng/certificates/")<br> )<br> );<br>};</i></font><br></div><div>And here is my netcat destination conf:</div><div><br></div><div><i>destination d_nc_tcp {<br> network(<br> "192.168.133.31"<br> port("7601")<br> template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA} ${MESSAGE}\n")<br> template-escape(no)<br> disk-buffer(<br> mem-buf-size(50M)<br> disk-buf-size(100M)<br> reliable(yes)<br> dir("/syslog-ng/log")<br> )<br> );<br>};</i><br><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">El mar., 18 dic. 2018 a las 10:35, Scheidler, Balázs (<<a href="mailto:balazs.scheidler@oneidentity.com" target="_blank">balazs.scheidler@oneidentity.com</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hmm.. are you sure it's not because of an embedded line-break character of some sorts? the template mechanism does not handle embedded NULs properly, but otherwise should work just fine with binary characters if they are already part of the message.</div><div><br></div><div>can you please do a hexdump of what syslog-ng sends to the relay node? could you please send me the binary file and the accompanying source statement?<br></div><div><br></div><div>thank you.<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 18, 2018 at 10:24 AM Jose Angel Santiago <<a href="mailto:jasantiago@stratio.com" target="_blank">jasantiago@stratio.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my special use case.<br></div><div><br></div><div>Unfortunately my syslog-ng agent reads some binary files which are piped through the log file I'm processing and sending it to elasticsearch. When this happens, the elastic bulk post fail (So I'am affected by this issue <a href="https://github.com/balabit/syslog-ng/issues/1993" target="_blank">https://github.com/balabit/syslog-ng/issues/1993</a>). I've debugged it and the problem was that message generated when applying the template is empty.</div><div><br></div><div>This is my network destination:<br></div><div><br></div><div>destination d_syslog_tcp {<br> network(<br></div><div> "syslog-relay.node"<br></div><div> port(7601)<br> template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n")<br> template-escape(no)<br> disk-buffer(<br> mem-buf-size(200M)<br> disk-buf-size(500M)<br> reliable(yes)<br> dir("/var/syslog-ng")<br> )<br> );<br>};<br></div><div><br></div>I've opened a netcat server as syslog-relay.node and piped a small binary file through the stdout file which I'm using as source, and this what the relay receives:</div><div dir="ltr"><br></div><div dir="ltr"><font size="1">2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 � <br>2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F<br>2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 <br> $<br>2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4) </font><br><br></div><div>As you can see, fourth message has no timestamp, hostname or ip address, it only contains "$"char, so when this message is sent to elastic, I've got a mapperException since it tries to validate the received timestamp.<br></div><div>I know I can customize elastic so it doesn't fail in case a wrong timestamp is received, but the point is that syslog-ng seems to fail when templating the message, doesn't it?</div><div><br></div><div>Also it would be great if syslog-ng had any mechanism to filter all those binary messages, I've already tried with filter expression, excluding all non-printable chars, but the issue still happens.</div><div><br></div><div>Regards.<br></div><div><br></div><div dir="ltr"><br></div><div dir="ltr"><div>-- <br><div dir="ltr" class="gmail-m_7558741252153678266gmail-m_4244783128874264723gmail-m_-3847052571697906050gmail-m_-1109524163579465255gmail-m_8747694410451953117gmail-m_2269236669992505383gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><span style="font-size:9pt;font-family:Tahoma;color:rgb(217,217,217);font-weight:700;vertical-align:baseline;white-space:pre-wrap">|</span><span style="font-size:9pt;font-family:Tahoma;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap"> Jose Angel Santiago</span><br></div><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.7273px"><span><br><p dir="ltr" style="line-height:1.656;margin-top:0pt;margin-bottom:0pt"><a href="http://www.stratio.com/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" alt="Logo_signature2.png" width="96" height="22"></span></a></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Vía de las dos Castillas, 33, Ática 4, 3ª Planta</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">28224 Pozuelo de Alarcón, Madrid, Spain </span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);vertical-align:baseline;white-space:pre-wrap">+34 918 286 473 | </span><a href="http://www.stratio.com" target="_blank"><span style="font-size:8pt;font-family:Tahoma;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap">www.stratio.com</span></a></p><a href="https://twitter.com/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" width="20" height="20"></span></a><a href="https://www.linkedin.com/company/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" width="20" height="20"></span></a><a href="https://www.youtube.com/c/StratioBD" target="_blank"><span style="font-size:8pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" width="20" height="20"></span></a></span><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_7558741252153678266gmail-m_4244783128874264723gmail-m_-3847052571697906050gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><span style="font-size:9pt;font-family:Tahoma;color:rgb(217,217,217);font-weight:700;vertical-align:baseline;white-space:pre-wrap">|</span><span style="font-size:9pt;font-family:Tahoma;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap"> Jose Angel Santiago</span><br></div><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.7273px"><span><br><p dir="ltr" style="line-height:1.656;margin-top:0pt;margin-bottom:0pt"><a href="http://www.stratio.com/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" alt="Logo_signature2.png" width="96" height="22"></span></a></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Vía de las dos Castillas, 33, Ática 4, 3ª Planta</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">28224 Pozuelo de Alarcón, Madrid, Spain </span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);vertical-align:baseline;white-space:pre-wrap">+34 918 286 473 | </span><a href="http://www.stratio.com" target="_blank"><span style="font-size:8pt;font-family:Tahoma;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap">www.stratio.com</span></a></p><a href="https://twitter.com/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" width="20" height="20"></span></a><a href="https://www.linkedin.com/company/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" width="20" height="20"></span></a><a href="https://www.youtube.com/c/StratioBD" target="_blank"><span style="font-size:8pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap"><img style="border: medium none;" width="20" height="20"></span></a></span><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><span style="font-size:9pt;font-family:Tahoma;color:rgb(217,217,217);font-weight:700;vertical-align:baseline;white-space:pre-wrap">|</span><span style="font-size:9pt;font-family:Tahoma;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap"> Jose Angel Santiago</span><br></div><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:12.7273px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.7273px"><span><br><p dir="ltr" style="line-height:1.656;margin-top:0pt;margin-bottom:0pt"><a href="http://www.stratio.com/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/SfSwo77PKD8TgM6em8B7mlcqOV9_N-AoAASRBMbZ3PFYgdFIibiMdr3zR_AZbMScWJOeRN7me-R_nK6vn1rnHSbApJVGfEjendjUa7LiGewC_fPGilVYciUS7E9v4mpKpl--caud" style="border: medium none;" alt="Logo_signature2.png" width="96" height="22"></span></a></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Vía de las dos Castillas, 33, Ática 4, 3ª Planta</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">28224 Pozuelo de Alarcón, Madrid, Spain </span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Tahoma;color:rgb(153,153,153);vertical-align:baseline;white-space:pre-wrap">+34 918 286 473 | </span><a href="http://www.stratio.com" target="_blank"><span style="font-size:8pt;font-family:Tahoma;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap">www.stratio.com</span></a></p><a href="https://twitter.com/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/hOlHqJK94rZ7nBo9gzYKhLiwgogX1sgXV81pPDpAHHyRVeCjHxw0THNCq19zhcZalZiYeVYt9r4T_7LhoeLMxN1eTMnG46IfttV83WkTGC3jL1z04craZ8mmUn9hNnxDTIgh4_cT" style="border: medium none;" width="20" height="20"></span></a><a href="https://www.linkedin.com/company/stratiobd" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/xs2vZh2SrmNnOaJO9i07vQzFNoghAaZytG_Zh09D9-ESjgOv5LHzLrbVNOFa-e3g5FYdmeg-kj6Ur6hID_h1joaSCdsHETfxTNqNSXan5nxBdGtmxq6NMWoh6puraVK1JGPxzhzX" style="border: medium none;" width="20" height="20"></span></a><a href="https://www.youtube.com/c/StratioBD" target="_blank"><span style="font-size:8pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh3.googleusercontent.com/GdwY5Qtr3qEaHfNZszPc0lGx52_bfO00F-ge1MzYQPNLQmXf2jO4z2kvWwIEKi2swQ-IfLjNvpS7o4wR0nwNhnhOFd7zZ1zJDtFVZLkp52XyaM0GpEXySD2iLbWq-dw0cmXUe7Jj" style="border: medium none;" width="20" height="20"></span></a></span><br></div></div></div></div></div></div></div></div></div></div></div></div></div>