[syslog-ng] ISODATE Macro not working when message contains non-ascii chars
Scheidler, Balázs
balazs.scheidler at oneidentity.com
Thu Dec 20 12:58:50 UTC 2018
If I understand everything right, then the problem is that you are reading
the output as a multi-line capable file source multi-line-mode(indented).
This means that your input file will contain messages that span multiple
lines. When you read that into syslog-ng, your $MSG macro will have an
embedded newline character in it.
With that said, you are using this template: *template("${ISODATE} ${HOST}
${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA} ${MESSAGE}\n")*
So this could produce a header (with ISODATE and everything else), and at
the end the $MESSAGE value, with NL characters in it.
So whenever you see an empty line, that's because of the value of $MESSAGE.
To confirm that this is the case, please enclose the ${MESSAGE} macro in
some visible characters, like this:
*template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA}
>>>${MESSAGE}<<<\n")*
You will see that the newline will be within >>> and <<<.
To resolve the situation please consider:
1) whether you actually need a multi-line capable source. By the look of
it, you are reading stdout files, line-by-line. Working in multi-line mode
means that a single message will span multiple input lines.
2) if you do (for instance to capture a multi-line backtrace as a single
message), you can always use "$(indent-multi-line $MESSAGE)" in place of
${MESSAGE} in the template above. That will insert whitespace into the
output file at NL barriers, so that records can still be reconstructed.
3) if you selectively need multi-line messages in one destination and not
in another, you can always use flags(no-multi-line) at the destination,
which will simply remove NL characters _after_ the template has been
formatted.
Also, you will find, that the template() mechanism is not capable of
working with embedded NUL characters, so if you happen to capture an
embedded NUL into a message, due to the multi-line mechanism, that can
terminate the template() based output somewhat earlier.
You can also remove invalid characters using the $(sanitize) template
function, e.g. $(sanitize $MESSAGE). $(sanitize) has the following syntax:
/*
* $(sanitize [opts] $arg1 $arg2 ...)
*
* Options:
* --ctrl-chars or -c Filter control characters (default)
* --no-ctrl-chars or -C Don't filter control characters
* --invalid-chars <set> or -i Set of characters to be translated,
default "/"
* --replacement <replace> or -r Single character replacement for invalid
chars.
*/
This removes all control characters, and the set specified in
--invalid-chars, which defaults to '/'.
On Thu, Dec 20, 2018 at 11:11 AM Jose Angel Santiago <jasantiago at stratio.com>
wrote:
> Hi,
> I redirected my netcat listener output to a file, and I don't see any
> blank lines on that file. So I decided to put a netcat as the final
> destination (i.e syslog-agent -> syslog-relay -> netcat) redirecting its
> output to a file.
>
> I've attached a sample of the the text plain file, where you can see a
> blank message (with would make elasticsearck bulk fail)
> I've also attached the hexdump of that file.
>
> I'm using the dqtool file as a binary file for my tests.
>
> Here is my source statement for syslog-agent:
>
>
>
>
>
>
>
>
>
>
>
>
> *source s_stdout_files { wildcard-file( base-dir("/logs")
> filename-pattern("stdout") max-files(100000) follow-freq(0.5)
> monitor-method(poll) log-fetch-limit(100) log-iw-size(15000)
> flags(no-parse) recursive(yes) multi-line-mode(indented)); };*
>
> Here is my source statement for syslog-relay:
>
>
>
>
>
>
>
>
>
>
>
> *source s_network { network( port(7601)
> transport("tls") flags(no-multi-line, no-parse) tls(
> key-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.key")
> cert-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.pem")
> ca-dir("/etc/syslog-ng/certificates/") ) );};*
> And here is my netcat destination conf:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *destination d_nc_tcp { network( "192.168.133.31"
> port("7601") template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE}
> ${TASK} ${SDATA} ${MESSAGE}\n") template-escape(no)
> disk-buffer( mem-buf-size(50M)
> disk-buf-size(100M) reliable(yes)
> dir("/syslog-ng/log") ) );};*
>
>
> El mar., 18 dic. 2018 a las 10:35, Scheidler, Balázs (<
> balazs.scheidler at oneidentity.com>) escribió:
>
>> Hmm.. are you sure it's not because of an embedded line-break character
>> of some sorts? the template mechanism does not handle embedded NULs
>> properly, but otherwise should work just fine with binary characters if
>> they are already part of the message.
>>
>> can you please do a hexdump of what syslog-ng sends to the relay node?
>> could you please send me the binary file and the accompanying source
>> statement?
>>
>> thank you.
>>
>> On Tue, Dec 18, 2018 at 10:24 AM Jose Angel Santiago <
>> jasantiago at stratio.com> wrote:
>>
>>> Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my
>>> special use case.
>>>
>>> Unfortunately my syslog-ng agent reads some binary files which are piped
>>> through the log file I'm processing and sending it to elasticsearch. When
>>> this happens, the elastic bulk post fail (So I'am affected by this issue
>>> https://github.com/balabit/syslog-ng/issues/1993). I've debugged it and
>>> the problem was that message generated when applying the template is empty.
>>>
>>> This is my network destination:
>>>
>>> destination d_syslog_tcp {
>>> network(
>>> "syslog-relay.node"
>>> port(7601)
>>> template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n")
>>> template-escape(no)
>>> disk-buffer(
>>> mem-buf-size(200M)
>>> disk-buf-size(500M)
>>> reliable(yes)
>>> dir("/var/syslog-ng")
>>> )
>>> );
>>> };
>>>
>>> I've opened a netcat server as syslog-relay.node and piped a small
>>> binary file through the stdout file which I'm using as source, and this
>>> what the relay receives:
>>>
>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 �
>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F
>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17
>>>
>>> $
>>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4)
>>>
>>> As you can see, fourth message has no timestamp, hostname or ip address,
>>> it only contains "$"char, so when this message is sent to elastic, I've got
>>> a mapperException since it tries to validate the received timestamp.
>>> I know I can customize elastic so it doesn't fail in case a wrong
>>> timestamp is received, but the point is that syslog-ng seems to fail when
>>> templating the message, doesn't it?
>>>
>>> Also it would be great if syslog-ng had any mechanism to filter all
>>> those binary messages, I've already tried with filter expression, excluding
>>> all non-printable chars, but the issue still happens.
>>>
>>> Regards.
>>>
>>>
>>> --
>>>
>>> | Jose Angel Santiago
>>>
>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>
>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>
>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>
>>> +34 918 286 473 | www.stratio.com
>>> <https://twitter.com/stratiobd>
>>> <https://www.linkedin.com/company/stratiobd>
>>> <https://www.youtube.com/c/StratioBD>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
> --
>
> | Jose Angel Santiago
>
> [image: Logo_signature2.png] <http://www.stratio.com/>
>
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>
> 28224 Pozuelo de Alarcón, Madrid, Spain
>
> +34 918 286 473 | www.stratio.com
> <https://twitter.com/stratiobd>
> <https://www.linkedin.com/company/stratiobd>
> <https://www.youtube.com/c/StratioBD>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181220/f0b7ee99/attachment-0001.html>
More information about the syslog-ng
mailing list