Hi, I redirected my netcat listener output to a file, and I don't see any blank lines on that file. So I decided to put a netcat as the final destination (i.e syslog-agent -> syslog-relay -> netcat) redirecting its output to a file. I've attached a sample of the the text plain file, where you can see a blank message (with would make elasticsearck bulk fail) I've also attached the hexdump of that file. I'm using the dqtool file as a binary file for my tests. Here is my source statement for syslog-agent: *source s_stdout_files { wildcard-file( base-dir("/logs") filename-pattern("stdout") max-files(100000) follow-freq(0.5) monitor-method(poll) log-fetch-limit(100) log-iw-size(15000) flags(no-parse) recursive(yes) multi-line-mode(indented)); };* Here is my source statement for syslog-relay: *source s_network { network( port(7601) transport("tls") flags(no-multi-line, no-parse) tls( key-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.key") cert-file("/etc/syslog-ng/certificates/syslog-relay-1.logging.pem") ca-dir("/etc/syslog-ng/certificates/") ) );};* And here is my netcat destination conf: *destination d_nc_tcp { network( "192.168.133.31" port("7601") template("${ISODATE} ${HOST} ${CONTAINER-IP} ${SERVICE} ${TASK} ${SDATA} ${MESSAGE}\n") template-escape(no) disk-buffer( mem-buf-size(50M) disk-buf-size(100M) reliable(yes) dir("/syslog-ng/log") ) );};* El mar., 18 dic. 2018 a las 10:35, Scheidler, Balázs (< balazs.scheidler@oneidentity.com>) escribió:
Hmm.. are you sure it's not because of an embedded line-break character of some sorts? the template mechanism does not handle embedded NULs properly, but otherwise should work just fine with binary characters if they are already part of the message.
can you please do a hexdump of what syslog-ng sends to the relay node? could you please send me the binary file and the accompanying source statement?
thank you.
On Tue, Dec 18, 2018 at 10:24 AM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my special use case.
Unfortunately my syslog-ng agent reads some binary files which are piped through the log file I'm processing and sending it to elasticsearch. When this happens, the elastic bulk post fail (So I'am affected by this issue https://github.com/balabit/syslog-ng/issues/1993). I've debugged it and the problem was that message generated when applying the template is empty.
This is my network destination:
destination d_syslog_tcp { network( "syslog-relay.node" port(7601) template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n") template-escape(no) disk-buffer( mem-buf-size(200M) disk-buf-size(500M) reliable(yes) dir("/var/syslog-ng") ) ); };
I've opened a netcat server as syslog-relay.node and piped a small binary file through the stdout file which I'm using as source, and this what the relay receives:
2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 � 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17
$ 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4)
As you can see, fourth message has no timestamp, hostname or ip address, it only contains "$"char, so when this message is sent to elastic, I've got a mapperException since it tries to validate the received timestamp. I know I can customize elastic so it doesn't fail in case a wrong timestamp is received, but the point is that syslog-ng seems to fail when templating the message, doesn't it?
Also it would be great if syslog-ng had any mechanism to filter all those binary messages, I've already tried with filter expression, excluding all non-printable chars, but the issue still happens.
Regards.
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>