Re: [syslog-ng] parsing JSON logs just won't work

18 Oct 2018

      Hi!

For a quick idea, you can start syslog-ng in debug mode where you can see
details about the message parsing. You will see if the parsing or the
template had problems.

You need to start syslog-ng with the -dv options to do that.

Regards,
Gábor

On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand@gmail.com>
wrote:
...
Hi,
I can’t get JSON parsing to work. I’ve consulted the documentation and
Google but with no luck.
I have an app, that puts out simple json log messages like:
{"level":"error","message":"connection ended without disconnect
receipt","timestamp":"2018-10-12T17:49:08.650Z"}
All I want to do, is parse these 3 values and send them to a hosted
Graylog cluster. Sending works, but the message gets inserted as
application name:   {"level"
    message:                    "error","message":"connection ended
without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
it's almost like syslog-ng doesn't even interpret the file as json. I
tried different variants  but I am at my wits end now...
This is my config (on the application host; it should send the logs
directly to the logging cluster)
@version: 3.5
    @include "scl.conf"
    @include "`scl-root`/system/tty10.conf"
options { chain_hostnames(off); flush_lines(0); use_dns(no);
use_fqdn(no);
           owner("root"); group("adm"); perm(0640); stats_freq(0);
           bad_hostname("^gconfd$");
    };
source s_src {
        file(
            “/var/log/worker/error.log"
            flags(no-parse)
            );
    };
template unitManagerTemplate {
        template("$(format-json --scope dot-nv-pairs) [sdid@123456
X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
    };
destination ovhPaaSLogs {
        tcp("gra2.logs.ovh.com"
            port(6514),
            template(unitManagerTemplate),
            ts_format("iso"),
            tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")),
            keep-alive(yes),
            so_keepalive(yes),
        );
    };
parser p_json {
        json-parser(prefix(".json."));
    };
log {
        source(s_src);
        parser(p_json);
        destination(ovhPaaSLogs);
    };
@include "/etc/syslog-ng/conf.d/"
I also tried a different a template variant like this:
template("${.json.level} ${.json.message} ${.json.timestamp}
[sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
I also tried parsing the messages as text:
template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"}
- [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\"
facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");
What shows up in Graylog is absolutely identical (like described in the
beginning). In fact, every variant that I tried changed absolutely nothing.
The conf.d folder is empty though.
I’d appreciate any help!
Best regards,
Michael
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Re: [syslog-ng] parsing JSON logs just won't work

Nagy, Gábor