[syslog-ng] ISODATE Macro not working when message contains non-ascii chars
Jose Angel Santiago
jasantiago at stratio.com
Thu Dec 20 16:39:59 UTC 2018
Using the indent-multi-line along with sanitize function, I've managed to
send every message perfectly parsed to elasticsearch.
Thank you so much for your excellent support!!
El jue., 20 dic. 2018 a las 13:59, Scheidler, Balázs (<
balazs.scheidler at oneidentity.com>) escribió:
> If I understand everything right, then the problem is that you are reading
> the output as a multi-line capable file source multi-line-mode(indented).
>
> This means that your input file will contain messages that span multiple
> lines. When you read that into syslog-ng, your $MSG macro will have an
> embedded newline character in it.
>
> With that said, you are using this template: *template("${ISODATE}
> ${HOST} ${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA} ${MESSAGE}\n")*
>
> So this could produce a header (with ISODATE and everything else), and at
> the end the $MESSAGE value, with NL characters in it.
>
> So whenever you see an empty line, that's because of the value of $MESSAGE.
>
> To confirm that this is the case, please enclose the ${MESSAGE} macro in
> some visible characters, like this:
>
> *template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA}
> >>>${MESSAGE}<<<\n")*
>
> You will see that the newline will be within >>> and <<<.
>
> To resolve the situation please consider:
> 1) whether you actually need a multi-line capable source. By the look of
> it, you are reading stdout files, line-by-line. Working in multi-line mode
> means that a single message will span multiple input lines.
> 2) if you do (for instance to capture a multi-line backtrace as a single
> message), you can always use "$(indent-multi-line $MESSAGE)" in place of
> ${MESSAGE} in the template above. That will insert whitespace into the
> output file at NL barriers, so that records can still be reconstructed.
> 3) if you selectively need multi-line messages in one destination and not
> in another, you can always use flags(no-multi-line) at the destination,
> which will simply remove NL characters _after_ the template has been
> formatted.
>
> Also, you will find, that the template() mechanism is not capable of
> working with embedded NUL characters, so if you happen to capture an
> embedded NUL into a message, due to the multi-line mechanism, that can
> terminate the template() based output somewhat earlier.
>
> You can also remove invalid characters using the $(sanitize) template
> function, e.g. $(sanitize $MESSAGE). $(sanitize) has the following syntax:
>
> /*
> * $(sanitize [opts] $arg1 $arg2 ...)
> *
> * Options:
> * --ctrl-chars or -c Filter control characters (default)
> * --no-ctrl-chars or -C Don't filter control characters
> * --invalid-chars <set> or -i Set of characters to be translated,
> default "/"
> * --replacement <replace> or -r Single character replacement for
> invalid chars.
> */
> This removes all control characters, and the set specified in
> --invalid-chars, which defaults to '/'.
>
>
> On Thu, Dec 20, 2018 at 11:11 AM Jose Angel Santiago <
> jasantiago at stratio.com> wrote:
>
>> Hi,
>> I redirected my netcat listener output to a file, and I don't see any
>> blank lines on that file. So I decided to put a netcat as the final
>> destination (i.e syslog-agent -> syslog-relay -> netcat) redirecting its
>> output to a file.
>>
>> I've attached a sample of the the text plain file, where you can see a
>> blank message (with would make elasticsearck bulk fail)
>> I've also attached the hexdump of that file.
>>
>> I'm using the dqtool file as a binary file for my tests.
>>
>> Here is my source statement for syslog-agent:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *source s_stdout_files { wildcard-file( base-dir("/logs")
>> filename-pattern("stdout") max-files(100000) follow-freq(0.5)
>> monitor-method(poll) log-fetch-limit(100) log-iw-size(15000)
>> flags(no-parse) recursive(yes) multi-line-mode(indented)); };*
>>
>> Here is my source statement for syslog-relay:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *source s_network { network( port(7601)
>> transport("tls") flags(no-multi-line, no-parse) tls(
>> key-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.key")
>> cert-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.pem")
>> ca-dir("/etc/syslog-ng/certificates/") ) );};*
>> And here is my netcat destination conf:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *destination d_nc_tcp { network( "192.168.133.31"
>> port("7601") template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE}
>> ${TASK} ${SDATA} ${MESSAGE}\n") template-escape(no)
>> disk-buffer( mem-buf-size(50M)
>> disk-buf-size(100M) reliable(yes)
>> dir("/syslog-ng/log") ) );};*
>>
>>
>> El mar., 18 dic. 2018 a las 10:35, Scheidler, Balázs (<
>> balazs.scheidler at oneidentity.com>) escribió:
>>
>>> Hmm.. are you sure it's not because of an embedded line-break character
>>> of some sorts? the template mechanism does not handle embedded NULs
>>> properly, but otherwise should work just fine with binary characters if
>>> they are already part of the message.
>>>
>>> can you please do a hexdump of what syslog-ng sends to the relay node?
>>> could you please send me the binary file and the accompanying source
>>> statement?
>>>
>>> thank you.
>>>
>>> On Tue, Dec 18, 2018 at 10:24 AM Jose Angel Santiago <
>>> jasantiago at stratio.com> wrote:
>>>
>>>> Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my
>>>> special use case.
>>>>
>>>> Unfortunately my syslog-ng agent reads some binary files which are
>>>> piped through the log file I'm processing and sending it to elasticsearch.
>>>> When this happens, the elastic bulk post fail (So I'am affected by this
>>>> issue https://github.com/balabit/syslog-ng/issues/1993). I've debugged
>>>> it and the problem was that message generated when applying the template is
>>>> empty.
>>>>
>>>> This is my network destination:
>>>>
>>>> destination d_syslog_tcp {
>>>> network(
>>>> "syslog-relay.node"
>>>> port(7601)
>>>> template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n")
>>>> template-escape(no)
>>>> disk-buffer(
>>>> mem-buf-size(200M)
>>>> disk-buf-size(500M)
>>>> reliable(yes)
>>>> dir("/var/syslog-ng")
>>>> )
>>>> );
>>>> };
>>>>
>>>> I've opened a netcat server as syslog-relay.node and piped a small
>>>> binary file through the stdout file which I'm using as source, and this
>>>> what the relay receives:
>>>>
>>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 �
>>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F
>>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17
>>>>
>>>> $
>>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4)
>>>>
>>>> As you can see, fourth message has no timestamp, hostname or ip
>>>> address, it only contains "$"char, so when this message is sent to elastic,
>>>> I've got a mapperException since it tries to validate the received
>>>> timestamp.
>>>> I know I can customize elastic so it doesn't fail in case a wrong
>>>> timestamp is received, but the point is that syslog-ng seems to fail when
>>>> templating the message, doesn't it?
>>>>
>>>> Also it would be great if syslog-ng had any mechanism to filter all
>>>> those binary messages, I've already tried with filter expression, excluding
>>>> all non-printable chars, but the issue still happens.
>>>>
>>>> Regards.
>>>>
>>>>
>>>> --
>>>>
>>>> | Jose Angel Santiago
>>>>
>>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>>
>>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>>
>>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>>
>>>> +34 918 286 473 | www.stratio.com
>>>> <https://twitter.com/stratiobd>
>>>> <https://www.linkedin.com/company/stratiobd>
>>>> <https://www.youtube.com/c/StratioBD>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>> --
>>
>> | Jose Angel Santiago
>>
>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>
>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>
>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>
>> +34 918 286 473 | www.stratio.com
>> <https://twitter.com/stratiobd>
>> <https://www.linkedin.com/company/stratiobd>
>> <https://www.youtube.com/c/StratioBD>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com
<https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd>
<https://www.youtube.com/c/StratioBD>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181220/9c41e71d/attachment-0001.html>
More information about the syslog-ng
mailing list