[syslog-ng] parsing JSON logs just won't work

17 Oct 2018

      Hi,

I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck.

I have an app, that puts out simple json log messages like:

    {"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}

All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as

    application name: 	{"level"
    message: 			"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}

it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants  but I am at my wits end now...

This is my config (on the application host; it should send the logs directly to the logging cluster)

    @version: 3.5
    @include "scl.conf"
    @include "`scl-root`/system/tty10.conf"

    options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
           owner("root"); group("adm"); perm(0640); stats_freq(0);
           bad_hostname("^gconfd$");
    };

    source s_src { 
        file(
            “/var/log/worker/error.log"
            flags(no-parse)
            );
    };

    template unitManagerTemplate {
        template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
    };

    destination ovhPaaSLogs {
        tcp("gra2.logs.ovh.com"
            port(6514),
            template(unitManagerTemplate),
            ts_format("iso"),
            tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")),
            keep-alive(yes),
            so_keepalive(yes),
        );
    };

    parser p_json { 
        json-parser(prefix(".json.")); 
    };

    log {
        source(s_src);
        parser(p_json);
        destination(ovhPaaSLogs);
    };

    @include "/etc/syslog-ng/conf.d/"

I also tried a different a template variant like this:

    template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);

I also tried parsing the messages as text:

    template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");

What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though.
I’d appreciate any help!

Best regards,

Michael

[syslog-ng] parsing JSON logs just won't work

Michael Niemand