<div dir="ltr">Thanks a lot for your answers. I realised I've got a super old version of syslog-ng. <br>I upgraded to the most recent version and after some minor tweaks to the config it works great now.<br>I'm sorry I didn't do this before writing to this list. Apologies!<br><br>Thanks again and best regards,<div>Michael</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 18, 2018 at 10:35 AM Szemere, László <<a href="mailto:laszlo.szemere@oneidentity.com">laszlo.szemere@oneidentity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Dear Michael,<div> thank you for the log message example! I putted it into a file called "message.json"</div><div> I had to strip down your config (not tls, no `scl-root`, etc.). After that I started up a syslog-ng with this minimal config:</div><div><br></div><div><div style="color:rgb(248,248,242);background-color:rgb(39,40,34);font-family:"Droid Sans Mono",monospace,monospace,"Droid Sans Fallback";font-weight:normal;font-size:14px;line-height:19px;white-space:pre-wrap"><div><span style="color:rgb(166,226,46)">@version</span><span style="color:rgb(248,248,240)">: 3.5</span></div><div><span style="color:rgb(166,226,46)">@include</span><span style="color:rgb(248,248,240)"> "scl.conf"</span></div><br><div><span style="color:rgb(248,248,242)">template unitManagerTemplate {</span></div><div><span style="color:rgb(248,248,242)"> template(</span><span style="color:rgb(230,219,116)">"$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=</span><span style="color:rgb(174,129,255)">\"</span><span style="color:rgb(230,219,116)">XXXXXXXXXXXXXXXXXXXXXXXXXX</span><span style="color:rgb(174,129,255)">\"\n</span><span style="color:rgb(230,219,116)">"</span><span style="color:rgb(248,248,242)">);</span></div><div><span style="color:rgb(248,248,242)">}</span><span style="color:rgb(248,248,240)">;</span></div><br><div><span style="color:rgb(248,248,242)">parser p_json { </span></div><div><span style="color:rgb(248,248,242)"> json</span><span style="color:rgb(249,38,114)">-</span><span style="color:rgb(248,248,242)">parser(prefix(</span><span style="color:rgb(230,219,116)">".json."</span><span style="color:rgb(248,248,242)">)); </span></div><div><span style="color:rgb(248,248,242)">}</span><span style="color:rgb(248,248,240)">;</span></div><br><div><span style="color:rgb(248,248,242)">destination ovhPaaSLogs {</span></div><div><span style="color:rgb(248,248,242)"> tcp(</span><span style="color:rgb(230,219,116)">"127.0.0.1"</span></div><div><span style="color:rgb(248,248,242)"> port(</span><span style="color:rgb(174,129,255)">6514</span><span style="color:rgb(248,248,242)">),</span></div><div><span style="color:rgb(248,248,242)"> template(unitManagerTemplate),</span></div><div><span style="color:rgb(248,248,242)"> );</span></div><div><span style="color:rgb(248,248,242)">}</span><span style="color:rgb(248,248,240)">;</span></div><br><div><span style="color:rgb(248,248,242)">source s_net {</span></div><div><span style="color:rgb(248,248,242)"> </span><span style="color:rgb(248,248,242)">file</span><span style="color:rgb(248,248,242)">(</span><span style="color:rgb(230,219,116)">"/source/message.json"</span></div><div><span style="color:rgb(248,248,242)"> flags(no</span><span style="color:rgb(249,38,114)">-</span><span style="color:rgb(248,248,242)">parse)</span></div><div><span style="color:rgb(248,248,242)"> );</span></div><div><span style="color:rgb(248,248,242)">}</span><span style="color:rgb(248,248,240)">;</span></div><br><div><span style="color:rgb(248,248,242)">log {</span></div><div><span style="color:rgb(248,248,242)"> source(s_net);</span></div><div><span style="color:rgb(248,248,242)"> parser(p_json);</span></div><div><span style="color:rgb(248,248,242)"> destination(ovhPaaSLogs);</span></div><div><span style="color:rgb(248,248,242)">}</span><span style="color:rgb(248,248,240)">;</span></div><br></div></div><div><br></div><div><br></div><div> And I got the following output: (started syslog-ng with the following command: /install/sbin/syslog-ng <b>-Fdev</b> -f /source/syslog-ng.conf)</div><div><br></div><div><div>[2018-10-18T08:27:49.450914] Incoming log entry; line='{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}'</div><div>[2018-10-18T08:27:49.451015] LogSource window is empty;</div><div>[2018-10-18T08:27:49.451076] Outgoing message; message='{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"\x0a'</div></div><div><br></div><div>The destination in an another window:</div><div><div><br></div><div>nc -kl 127.0.0.1 6514</div></div><div><div>{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"</div></div><div><br></div><div><br></div><div>note: The paths (/install, /source, etc.) are coming from the fact I am using dbld (<a href="https://github.com/balabit/syslog-ng/tree/master/dbld" target="_blank">https://github.com/balabit/syslog-ng/tree/master/dbld</a>), you might want to try it, to eliminate any environment effects. If it works, than we can compare your environment with dbld for differences.</div><div><br></div><div>Br,</div><div>Laci</div><div><br></div><div><br></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 18, 2018 at 8:18 AM, Nagy, Gábor <span dir="ltr"><<a href="mailto:gabor.nagy@oneidentity.com" target="_blank">gabor.nagy@oneidentity.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>Hi!<div dir="auto"><br></div><div dir="auto">For a quick idea, you can start syslog-ng in debug mode where you can see details about the message parsing. You will see if the parsing or the template had problems.</div><div dir="auto"><br></div><div dir="auto">You need to start syslog-ng with the -dv options to do that.</div><br>Regards,</div><div dir="auto">Gábor</div><div><div class="m_-6234555131916969734h5"><div dir="auto"><br><div class="gmail_quote" dir="auto"><div dir="ltr">On Wed, 17 Oct 2018, 18:05 Michael Niemand, <<a href="mailto:michael.niemand@gmail.com" target="_blank">michael.niemand@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck.<br>
<br>
I have an app, that puts out simple json log messages like:<br>
<br>
{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}<br>
<br>
All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as<br>
<br>
application name: {"level"<br>
message: "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}<br>
<br>
it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants but I am at my wits end now...<br>
<br>
This is my config (on the application host; it should send the logs directly to the logging cluster)<br>
<br>
@version: 3.5<br>
@include "scl.conf"<br>
@include "`scl-root`/system/tty10.conf"<br>
<br>
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);<br>
owner("root"); group("adm"); perm(0640); stats_freq(0);<br>
bad_hostname("^gconfd$");<br>
};<br>
<br>
source s_src { <br>
file(<br>
“/var/log/worker/error.log"<br>
flags(no-parse)<br>
);<br>
};<br>
<br>
template unitManagerTemplate {<br>
template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");<br>
};<br>
<br>
destination ovhPaaSLogs {<br>
tcp("<a href="http://gra2.logs.ovh.com" rel="noreferrer noreferrer" target="_blank">gra2.logs.ovh.com</a>"<br>
port(6514),<br>
template(unitManagerTemplate),<br>
ts_format("iso"),<br>
tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")),<br>
keep-alive(yes),<br>
so_keepalive(yes),<br>
);<br>
};<br>
<br>
parser p_json { <br>
json-parser(prefix(".json.")); <br>
};<br>
<br>
log {<br>
source(s_src);<br>
parser(p_json);<br>
destination(ovhPaaSLogs);<br>
};<br>
<br>
@include "/etc/syslog-ng/conf.d/"<br>
<br>
<br>
I also tried a different a template variant like this:<br>
<br>
template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);<br>
<br>
I also tried parsing the messages as text:<br>
<br>
template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");<br>
<br>
What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though.<br>
I’d appreciate any help!<br>
<br>
<br>
Best regards,<br>
<br>
Michael<br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div></div></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">LargePrefPlaceholder-XKUz1MEJBwkOM</div>