[syslog-ng] ISODATE Macro not working when message contains non-ascii chars

[Jose Angel Santiago]

Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my
special use case.

Unfortunately my syslog-ng agent reads some binary files which are piped
through the log file I'm processing and sending it to elasticsearch. When
this happens, the elastic bulk post fail (So I'am affected by this issue
https://github.com/balabit/syslog-ng/issues/1993). I've debugged it and the
problem was that message generated when applying the template is empty.

This is my network destination:

destination d_syslog_tcp {
    network(
        "syslog-relay.node"
        port(7601)
        template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n")
        template-escape(no)
        disk-buffer(
            mem-buf-size(200M)
            disk-buf-size(500M)
            reliable(yes)
            dir("/var/syslog-ng")
        )
    );
};

I've opened a netcat server as syslog-relay.node and piped a small binary
file through the stdout file which I'm using as source, and this what the
relay receives:

2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 �
2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F
2018-12-17T15:35:57+00:00 agent-4 192.168.54.17

$
2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4)

As you can see, fourth message has no timestamp, hostname or ip address, it
only contains "$"char, so when this message is sent to elastic, I've got a
mapperException since it tries to validate the received timestamp.
I know I can customize elastic so it doesn't fail in case a wrong timestamp
is received, but the point is that syslog-ng seems to fail when templating
the message, doesn't it?

Also it would be great if syslog-ng had any mechanism to filter all those
binary messages, I've already tried with filter expression, excluding all
non-printable chars, but the issue still happens.

Regards.


-- 

| Jose Angel Santiago

[image: Logo_signature2.png] <http://www.stratio.com/>

Vía de las dos Castillas, 33, Ática 4, 3ª Planta

28224 Pozuelo de Alarcón, Madrid, Spain

+34 918 286 473 | www.stratio.com
<https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd>
<https://www.youtube.com/c/StratioBD>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181218/f22f60e5/attachment.html>