[syslog-ng] parsing JSON logs just won't work

[Nagy, Gábor]

Hi!

For a quick idea, you can start syslog-ng in debug mode where you can see
details about the message parsing. You will see if the parsing or the
template had problems.

You need to start syslog-ng with the -dv options to do that.

Regards,
Gábor

On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand at gmail.com>
wrote:

> Hi,
>
> I can’t get JSON parsing to work. I’ve consulted the documentation and
> Google but with no luck.
>
> I have an app, that puts out simple json log messages like:
>
>     {"level":"error","message":"connection ended without disconnect
> receipt","timestamp":"2018-10-12T17:49:08.650Z"}
>
> All I want to do, is parse these 3 values and send them to a hosted
> Graylog cluster. Sending works, but the message gets inserted as
>
>     application name:   {"level"
>     message:                    "error","message":"connection ended
> without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
>
> it's almost like syslog-ng doesn't even interpret the file as json. I
> tried different variants  but I am at my wits end now...
>
> This is my config (on the application host; it should send the logs
> directly to the logging cluster)
>
>     @version: 3.5
>     @include "scl.conf"
>     @include "`scl-root`/system/tty10.conf"
>
>     options { chain_hostnames(off); flush_lines(0); use_dns(no);
> use_fqdn(no);
>            owner("root"); group("adm"); perm(0640); stats_freq(0);
>            bad_hostname("^gconfd$");
>     };
>
>     source s_src {
>         file(
>             “/var/log/worker/error.log"
>             flags(no-parse)
>             );
>     };
>
>     template unitManagerTemplate {
>         template("$(format-json --scope dot-nv-pairs) [sdid at 123456
> X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
>     };
>
>     destination ovhPaaSLogs {
>         tcp("gra2.logs.ovh.com"
>             port(6514),
>             template(unitManagerTemplate),
>             ts_format("iso"),
>             tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")),
>             keep-alive(yes),
>             so_keepalive(yes),
>         );
>     };
>
>     parser p_json {
>         json-parser(prefix(".json."));
>     };
>
>     log {
>         source(s_src);
>         parser(p_json);
>         destination(ovhPaaSLogs);
>     };
>
>     @include "/etc/syslog-ng/conf.d/"
>
>
> I also tried a different a template variant like this:
>
>     template("${.json.level} ${.json.message} ${.json.timestamp}
> [sdid at 123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
>
> I also tried parsing the messages as text:
>
>
> template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"}
> - [sdid at 32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\"
> facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");
>
> What shows up in Graylog is absolutely identical (like described in the
> beginning). In fact, every variant that I tried changed absolutely nothing.
> The conf.d folder is empty though.
> I’d appreciate any help!
>
>
> Best regards,
>
> Michael
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181018/b00a2c47/attachment.html>