[syslog-ng] ISODATE Macro not working when message contains non-ascii chars
Jose Angel Santiago
jasantiago at stratio.com
Thu Dec 20 10:11:28 UTC 2018
Hi,
I redirected my netcat listener output to a file, and I don't see any blank
lines on that file. So I decided to put a netcat as the final destination
(i.e syslog-agent -> syslog-relay -> netcat) redirecting its output to a
file.
I've attached a sample of the the text plain file, where you can see a
blank message (with would make elasticsearck bulk fail)
I've also attached the hexdump of that file.
I'm using the dqtool file as a binary file for my tests.
Here is my source statement for syslog-agent:
*source s_stdout_files { wildcard-file( base-dir("/logs")
filename-pattern("stdout") max-files(100000) follow-freq(0.5)
monitor-method(poll) log-fetch-limit(100) log-iw-size(15000)
flags(no-parse) recursive(yes) multi-line-mode(indented)); };*
Here is my source statement for syslog-relay:
*source s_network { network( port(7601)
transport("tls") flags(no-multi-line, no-parse) tls(
key-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.key")
cert-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.pem")
ca-dir("/etc/syslog-ng/certificates/") ) );};*
And here is my netcat destination conf:
*destination d_nc_tcp { network( "192.168.133.31"
port("7601") template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE}
${TASK} ${SDATA} ${MESSAGE}\n") template-escape(no)
disk-buffer( mem-buf-size(50M)
disk-buf-size(100M) reliable(yes)
dir("/syslog-ng/log") ) );};*
El mar., 18 dic. 2018 a las 10:35, Scheidler, Balázs (<
balazs.scheidler at oneidentity.com>) escribió:
> Hmm.. are you sure it's not because of an embedded line-break character of
> some sorts? the template mechanism does not handle embedded NULs properly,
> but otherwise should work just fine with binary characters if they are
> already part of the message.
>
> can you please do a hexdump of what syslog-ng sends to the relay node?
> could you please send me the binary file and the accompanying source
> statement?
>
> thank you.
>
> On Tue, Dec 18, 2018 at 10:24 AM Jose Angel Santiago <
> jasantiago at stratio.com> wrote:
>
>> Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my
>> special use case.
>>
>> Unfortunately my syslog-ng agent reads some binary files which are piped
>> through the log file I'm processing and sending it to elasticsearch. When
>> this happens, the elastic bulk post fail (So I'am affected by this issue
>> https://github.com/balabit/syslog-ng/issues/1993). I've debugged it and
>> the problem was that message generated when applying the template is empty.
>>
>> This is my network destination:
>>
>> destination d_syslog_tcp {
>> network(
>> "syslog-relay.node"
>> port(7601)
>> template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n")
>> template-escape(no)
>> disk-buffer(
>> mem-buf-size(200M)
>> disk-buf-size(500M)
>> reliable(yes)
>> dir("/var/syslog-ng")
>> )
>> );
>> };
>>
>> I've opened a netcat server as syslog-relay.node and piped a small binary
>> file through the stdout file which I'm using as source, and this what the
>> relay receives:
>>
>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 �
>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F
>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17
>>
>> $
>> 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4)
>>
>> As you can see, fourth message has no timestamp, hostname or ip address,
>> it only contains "$"char, so when this message is sent to elastic, I've got
>> a mapperException since it tries to validate the received timestamp.
>> I know I can customize elastic so it doesn't fail in case a wrong
>> timestamp is received, but the point is that syslog-ng seems to fail when
>> templating the message, doesn't it?
>>
>> Also it would be great if syslog-ng had any mechanism to filter all those
>> binary messages, I've already tried with filter expression, excluding all
>> non-printable chars, but the issue still happens.
>>
>> Regards.
>>
>>
>> --
>>
>> | Jose Angel Santiago
>>
>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>
>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>
>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>
>> +34 918 286 473 | www.stratio.com
>> <https://twitter.com/stratiobd>
>> <https://www.linkedin.com/company/stratiobd>
>> <https://www.youtube.com/c/StratioBD>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com
<https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd>
<https://www.youtube.com/c/StratioBD>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181220/cdf13744/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hexpartial_logs
Type: application/octet-stream
Size: 9476 bytes
Desc: not available
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181220/cdf13744/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: partial_logs
Type: application/octet-stream
Size: 1907 bytes
Desc: not available
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181220/cdf13744/attachment-0003.obj>
More information about the syslog-ng
mailing list