parsing JSON logs just won't work
Hi, I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck. I have an app, that puts out simple json log messages like: {"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"} All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as application name: {"level" message: "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"} it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants but I am at my wits end now... This is my config (on the application host; it should send the logs directly to the logging cluster) @version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf" options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; source s_src { file( “/var/log/worker/error.log" flags(no-parse) ); }; template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); }; destination ovhPaaSLogs { tcp("gra2.logs.ovh.com" port(6514), template(unitManagerTemplate), ts_format("iso"), tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")), keep-alive(yes), so_keepalive(yes), ); }; parser p_json { json-parser(prefix(".json.")); }; log { source(s_src); parser(p_json); destination(ovhPaaSLogs); }; @include "/etc/syslog-ng/conf.d/" I also tried a different a template variant like this: template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”); I also tried parsing the messages as text: template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n"); What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though. I’d appreciate any help! Best regards, Michael
Hi! For a quick idea, you can start syslog-ng in debug mode where you can see details about the message parsing. You will see if the parsing or the template had problems. You need to start syslog-ng with the -dv options to do that. Regards, Gábor On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand@gmail.com> wrote:
Hi,
I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck.
I have an app, that puts out simple json log messages like:
{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as
application name: {"level" message: "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants but I am at my wits end now...
This is my config (on the application host; it should send the logs directly to the logging cluster)
@version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf"
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); };
source s_src { file( “/var/log/worker/error.log" flags(no-parse) ); };
template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); };
destination ovhPaaSLogs { tcp("gra2.logs.ovh.com" port(6514), template(unitManagerTemplate), ts_format("iso"), tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")), keep-alive(yes), so_keepalive(yes), ); };
parser p_json { json-parser(prefix(".json.")); };
log { source(s_src); parser(p_json); destination(ovhPaaSLogs); };
@include "/etc/syslog-ng/conf.d/"
I also tried a different a template variant like this:
template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
I also tried parsing the messages as text:
template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");
What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though. I’d appreciate any help!
Best regards,
Michael
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Dear Michael, thank you for the log message example! I putted it into a file called "message.json" I had to strip down your config (not tls, no `scl-root`, etc.). After that I started up a syslog-ng with this minimal config: @version: 3.5 @include "scl.conf" template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\" XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); }; parser p_json { json-parser(prefix(".json.")); }; destination ovhPaaSLogs { tcp("127.0.0.1" port(6514), template(unitManagerTemplate), ); }; source s_net { file("/source/message.json" flags(no-parse) ); }; log { source(s_net); parser(p_json); destination(ovhPaaSLogs); }; And I got the following output: (started syslog-ng with the following command: /install/sbin/syslog-ng *-Fdev* -f /source/syslog-ng.conf) [2018-10-18T08:27:49.450914] Incoming log entry; line='{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}' [2018-10-18T08:27:49.451015] LogSource window is empty; [2018-10-18T08:27:49.451076] Outgoing message; message='{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"\x0a' The destination in an another window: nc -kl 127.0.0.1 6514 {"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX" note: The paths (/install, /source, etc.) are coming from the fact I am using dbld (https://github.com/balabit/syslog-ng/tree/master/dbld), you might want to try it, to eliminate any environment effects. If it works, than we can compare your environment with dbld for differences. Br, Laci On Thu, Oct 18, 2018 at 8:18 AM, Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi!
For a quick idea, you can start syslog-ng in debug mode where you can see details about the message parsing. You will see if the parsing or the template had problems.
You need to start syslog-ng with the -dv options to do that.
Regards, Gábor
On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand@gmail.com> wrote:
Hi,
I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck.
I have an app, that puts out simple json log messages like:
{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as
application name: {"level" message: "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants but I am at my wits end now...
This is my config (on the application host; it should send the logs directly to the logging cluster)
@version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf"
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); };
source s_src { file( “/var/log/worker/error.log" flags(no-parse) ); };
template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); };
destination ovhPaaSLogs { tcp("gra2.logs.ovh.com" port(6514), template(unitManagerTemplate), ts_format("iso"), tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")), keep-alive(yes), so_keepalive(yes), ); };
parser p_json { json-parser(prefix(".json.")); };
log { source(s_src); parser(p_json); destination(ovhPaaSLogs); };
@include "/etc/syslog-ng/conf.d/"
I also tried a different a template variant like this:
template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
I also tried parsing the messages as text:
template("{\"level\":\"${PRIORITY}\",\"message\":\"${ MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");
What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though. I’d appreciate any help!
Best regards,
Michael
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks a lot for your answers. I realised I've got a super old version of syslog-ng. I upgraded to the most recent version and after some minor tweaks to the config it works great now. I'm sorry I didn't do this before writing to this list. Apologies! Thanks again and best regards, Michael On Thu, Oct 18, 2018 at 10:35 AM Szemere, László < laszlo.szemere@oneidentity.com> wrote:
Dear Michael, thank you for the log message example! I putted it into a file called "message.json" I had to strip down your config (not tls, no `scl-root`, etc.). After that I started up a syslog-ng with this minimal config:
@version: 3.5 @include "scl.conf"
template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\" XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); };
parser p_json { json-parser(prefix(".json.")); };
destination ovhPaaSLogs { tcp("127.0.0.1" port(6514), template(unitManagerTemplate), ); };
source s_net { file("/source/message.json" flags(no-parse) ); };
log { source(s_net); parser(p_json); destination(ovhPaaSLogs); };
And I got the following output: (started syslog-ng with the following command: /install/sbin/syslog-ng *-Fdev* -f /source/syslog-ng.conf)
[2018-10-18T08:27:49.450914] Incoming log entry; line='{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}' [2018-10-18T08:27:49.451015] LogSource window is empty; [2018-10-18T08:27:49.451076] Outgoing message; message='{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"\x0a'
The destination in an another window:
nc -kl 127.0.0.1 6514 {"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"
note: The paths (/install, /source, etc.) are coming from the fact I am using dbld (https://github.com/balabit/syslog-ng/tree/master/dbld), you might want to try it, to eliminate any environment effects. If it works, than we can compare your environment with dbld for differences.
Br, Laci
On Thu, Oct 18, 2018 at 8:18 AM, Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi!
For a quick idea, you can start syslog-ng in debug mode where you can see details about the message parsing. You will see if the parsing or the template had problems.
You need to start syslog-ng with the -dv options to do that.
Regards, Gábor
On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand@gmail.com> wrote:
Hi,
I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck.
I have an app, that puts out simple json log messages like:
{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as
application name: {"level" message: "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants but I am at my wits end now...
This is my config (on the application host; it should send the logs directly to the logging cluster)
@version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf"
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); };
source s_src { file( “/var/log/worker/error.log" flags(no-parse) ); };
template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); };
destination ovhPaaSLogs { tcp("gra2.logs.ovh.com" port(6514), template(unitManagerTemplate), ts_format("iso"), tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")), keep-alive(yes), so_keepalive(yes), ); };
parser p_json { json-parser(prefix(".json.")); };
log { source(s_src); parser(p_json); destination(ovhPaaSLogs); };
@include "/etc/syslog-ng/conf.d/"
I also tried a different a template variant like this:
template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
I also tried parsing the messages as text:
template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");
What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though. I’d appreciate any help!
Best regards,
Michael
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- LargePrefPlaceholder-XKUz1MEJBwkOM
participants (3)
-
Michael Niemand
-
Nagy, Gábor
-
Szemere, László