Dear Michael,

 thank you for the log message example! I putted it into a file called "message.json"

 I had to strip down your config (not tls, no `scl-root`, etc.). After that I started up a syslog-ng with this minimal config:

@version: 3.5

@include "scl.conf"

template unitManagerTemplate {

template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");

};

parser p_json {

json-parser(prefix(".json."));

};

destination ovhPaaSLogs {

tcp("127.0.0.1"

port(6514),

template(unitManagerTemplate),

);

};

source s_net {

file("/source/message.json"

flags(no-parse)

);

};

log {

source(s_net);

parser(p_json);

destination(ovhPaaSLogs);

};

 And I got the following output: (started syslog-ng with the following command: /install/sbin/syslog-ng -Fdev -f /source/syslog-ng.conf)

[2018-10-18T08:27:49.450914] Incoming log entry; line='{"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}'

[2018-10-18T08:27:49.451015] LogSource window is empty;

[2018-10-18T08:27:49.451076] Outgoing message; message='{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"\x0a'

The destination in an another window:

nc -kl 127.0.0.1 6514

{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection ended without disconnect receipt","level":"error"}} [sdid@123456 X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"

note: The paths (/install, /source, etc.) are coming from the fact I am using dbld (https://github.com/balabit/syslog-ng/tree/master/dbld), you might want to try it, to eliminate any environment effects. If it works, than we can compare your environment with dbld for differences.

Br,

Laci

On Thu, Oct 18, 2018 at 8:18 AM, Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:

Hi!

For a quick idea, you can start syslog-ng in debug mode where you can see details about the message parsing. You will see if the parsing or the template had problems.

You need to start syslog-ng with the -dv options to do that.

Regards,

Gábor

On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand@gmail.com> wrote:

Hi,

I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck.

I have an app, that puts out simple json log messages like:

    {"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}

All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as

    application name:   {"level"
    message:                    "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}

it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants  but I am at my wits end now...

This is my config (on the application host; it should send the logs directly to the logging cluster)

    @version: 3.5
    @include "scl.conf"
    @include "`scl-root`/system/tty10.conf"

    options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
           owner("root"); group("adm"); perm(0640); stats_freq(0);
           bad_hostname("^gconfd$");
    };

    source s_src {
        file(
            “/var/log/worker/error.log"
            flags(no-parse)
            );
    };

    template unitManagerTemplate {
        template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
    };

    destination ovhPaaSLogs {
        tcp("gra2.logs.ovh.com"
            port(6514),
            template(unitManagerTemplate),
            ts_format("iso"),
            tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")),
            keep-alive(yes),
            so_keepalive(yes),
        );
    };

    parser p_json {
        json-parser(prefix(".json."));
    };

    log {
        source(s_src);
        parser(p_json);
        destination(ovhPaaSLogs);
    };

    @include "/etc/syslog-ng/conf.d/"


I also tried a different a template variant like this:

    template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);

I also tried parsing the messages as text:

    template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");

What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though.
I’d appreciate any help!


Best regards,

Michael

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq