Re: [syslog-ng] parsing JSON logs just won't work

18 Oct 2018

      Dear Michael,
 thank you for the log message example! I putted it into a file called
"message.json"
 I had to strip down your config (not tls, no `scl-root`, etc.). After that
I started up a syslog-ng with this minimal config:

@version: 3.5
@include "scl.conf"

template unitManagerTemplate {
template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"
XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
};

parser p_json {
json-parser(prefix(".json."));
};

destination ovhPaaSLogs {
tcp("127.0.0.1"
port(6514),
template(unitManagerTemplate),
);
};

source s_net {
file("/source/message.json"
flags(no-parse)
);
};

log {
source(s_net);
parser(p_json);
destination(ovhPaaSLogs);
};

 And I got the following output: (started syslog-ng with the following
command: /install/sbin/syslog-ng *-Fdev* -f /source/syslog-ng.conf)

[2018-10-18T08:27:49.450914] Incoming log entry;
line='{"level":"error","message":"connection ended without disconnect
receipt","timestamp":"2018-10-12T17:49:08.650Z"}'
[2018-10-18T08:27:49.451015] LogSource window is empty;
[2018-10-18T08:27:49.451076] Outgoing message;
message='{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection
ended without disconnect receipt","level":"error"}} [sdid@123456
X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"\x0a'

The destination in an another window:

nc -kl 127.0.0.1 6514
{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection
ended without disconnect receipt","level":"error"}} [sdid@123456
X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"

note: The paths (/install, /source, etc.) are coming from the fact I am
using dbld (https://github.com/balabit/syslog-ng/tree/master/dbld), you
might want to try it, to eliminate any environment effects. If it works,
than we can compare your environment with dbld for differences.

Br,
Laci

On Thu, Oct 18, 2018 at 8:18 AM, Nagy, Gábor <gabor.nagy@oneidentity.com>
wrote:
...
Hi!
For a quick idea, you can start syslog-ng in debug mode where you can see
details about the message parsing. You will see if the parsing or the
template had problems.
You need to start syslog-ng with the -dv options to do that.
Regards,
Gábor
On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand@gmail.com>
wrote:
...
Hi,
I can’t get JSON parsing to work. I’ve consulted the documentation and
Google but with no luck.
I have an app, that puts out simple json log messages like:
{"level":"error","message":"connection ended without disconnect
receipt","timestamp":"2018-10-12T17:49:08.650Z"}
All I want to do, is parse these 3 values and send them to a hosted
Graylog cluster. Sending works, but the message gets inserted as
application name:   {"level"
    message:                    "error","message":"connection ended
without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
it's almost like syslog-ng doesn't even interpret the file as json. I
tried different variants  but I am at my wits end now...
This is my config (on the application host; it should send the logs
directly to the logging cluster)
@version: 3.5
    @include "scl.conf"
    @include "`scl-root`/system/tty10.conf"
options { chain_hostnames(off); flush_lines(0); use_dns(no);
use_fqdn(no);
           owner("root"); group("adm"); perm(0640); stats_freq(0);
           bad_hostname("^gconfd$");
    };
source s_src {
        file(
            “/var/log/worker/error.log"
            flags(no-parse)
            );
    };
template unitManagerTemplate {
        template("$(format-json --scope dot-nv-pairs) [sdid@123456
X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
    };
destination ovhPaaSLogs {
        tcp("gra2.logs.ovh.com"
            port(6514),
            template(unitManagerTemplate),
            ts_format("iso"),
            tls(peer-verify("require-trusted")
ca_dir("/etc/ssl/certs/")),
            keep-alive(yes),
            so_keepalive(yes),
        );
    };
parser p_json {
        json-parser(prefix(".json."));
    };
log {
        source(s_src);
        parser(p_json);
        destination(ovhPaaSLogs);
    };
@include "/etc/syslog-ng/conf.d/"
I also tried a different a template variant like this:
template("${.json.level} ${.json.message} ${.json.timestamp}
[sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
I also tried parsing the messages as text:
template("{\"level\":\"${PRIORITY}\",\"message\":\"${
MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"
pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"]
${MSG}\n");
What shows up in Graylog is absolutely identical (like described in the
beginning). In fact, every variant that I tried changed absolutely nothing.
The conf.d folder is empty though.
I’d appreciate any help!
Best regards,
Michael
____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?
product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?
product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Re: [syslog-ng] parsing JSON logs just won't work

Szemere, László