[syslog-ng] parsing JSON logs just won't work

Thu Oct 18 08:35:47 UTC 2018

Dear Michael,
 thank you for the log message example! I putted it into a file called
"message.json"
 I had to strip down your config (not tls, no `scl-root`, etc.). After that
I started up a syslog-ng with this minimal config:

@version: 3.5
@include "scl.conf"

template unitManagerTemplate {
template("$(format-json --scope dot-nv-pairs) [sdid at 123456 X-OVH-TOKEN=\"
XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
};

parser p_json {
json-parser(prefix(".json."));
};

destination ovhPaaSLogs {
tcp("127.0.0.1"
port(6514),
template(unitManagerTemplate),
);
};

source s_net {
file("/source/message.json"
flags(no-parse)
);
};

log {
source(s_net);
parser(p_json);
destination(ovhPaaSLogs);
};

 And I got the following output: (started syslog-ng with the following
command: /install/sbin/syslog-ng *-Fdev* -f /source/syslog-ng.conf)

[2018-10-18T08:27:49.450914] Incoming log entry;
line='{"level":"error","message":"connection ended without disconnect
receipt","timestamp":"2018-10-12T17:49:08.650Z"}'
[2018-10-18T08:27:49.451015] LogSource window is empty;
[2018-10-18T08:27:49.451076] Outgoing message;
message='{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection
ended without disconnect receipt","level":"error"}} [sdid at 123456
X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"\x0a'

The destination in an another window:

nc -kl 127.0.0.1 6514
{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection
ended without disconnect receipt","level":"error"}} [sdid at 123456
X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"

note: The paths (/install, /source, etc.) are coming from the fact I am
using dbld (https://github.com/balabit/syslog-ng/tree/master/dbld), you
might want to try it, to eliminate any environment effects. If it works,
than we can compare your environment with dbld for differences.

Br,
Laci

On Thu, Oct 18, 2018 at 8:18 AM, Nagy, Gábor <gabor.nagy at oneidentity.com>
wrote:

> Hi!
>
> For a quick idea, you can start syslog-ng in debug mode where you can see
> details about the message parsing. You will see if the parsing or the
> template had problems.
>
> You need to start syslog-ng with the -dv options to do that.
>
> Regards,
> Gábor
>
> On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand at gmail.com>
> wrote:
>
>> Hi,
>>
>> I can’t get JSON parsing to work. I’ve consulted the documentation and
>> Google but with no luck.
>>
>> I have an app, that puts out simple json log messages like:
>>
>>     {"level":"error","message":"connection ended without disconnect
>> receipt","timestamp":"2018-10-12T17:49:08.650Z"}
>>
>> All I want to do, is parse these 3 values and send them to a hosted
>> Graylog cluster. Sending works, but the message gets inserted as
>>
>>     application name:   {"level"
>>     message:                    "error","message":"connection ended
>> without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
>>
>> it's almost like syslog-ng doesn't even interpret the file as json. I
>> tried different variants  but I am at my wits end now...
>>
>> This is my config (on the application host; it should send the logs
>> directly to the logging cluster)
>>
>>     @version: 3.5
>>     @include "scl.conf"
>>     @include "`scl-root`/system/tty10.conf"
>>
>>     options { chain_hostnames(off); flush_lines(0); use_dns(no);
>> use_fqdn(no);
>>            owner("root"); group("adm"); perm(0640); stats_freq(0);
>>            bad_hostname("^gconfd$");
>>     };
>>
>>     source s_src {
>>         file(
>>             “/var/log/worker/error.log"
>>             flags(no-parse)
>>             );
>>     };
>>
>>     template unitManagerTemplate {
>>         template("$(format-json --scope dot-nv-pairs) [sdid at 123456
>> X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
>>     };
>>
>>     destination ovhPaaSLogs {
>>         tcp("gra2.logs.ovh.com"
>>             port(6514),
>>             template(unitManagerTemplate),
>>             ts_format("iso"),
>>             tls(peer-verify("require-trusted")
>> ca_dir("/etc/ssl/certs/")),
>>             keep-alive(yes),
>>             so_keepalive(yes),
>>         );
>>     };
>>
>>     parser p_json {
>>         json-parser(prefix(".json."));
>>     };
>>
>>     log {
>>         source(s_src);
>>         parser(p_json);
>>         destination(ovhPaaSLogs);
>>     };
>>
>>     @include "/etc/syslog-ng/conf.d/"
>>
>>
>> I also tried a different a template variant like this:
>>
>>     template("${.json.level} ${.json.message} ${.json.timestamp}
>> [sdid at 123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
>>
>> I also tried parsing the messages as text:
>>
>>     template("{\"level\":\"${PRIORITY}\",\"message\":\"${
>> MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid at 32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"
>> pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"]
>> ${MSG}\n");
>>
>> What shows up in Graylog is absolutely identical (like described in the
>> beginning). In fact, every variant that I tried changed absolutely nothing.
>> The conf.d folder is empty though.
>> I’d appreciate any help!
>>
>>
>> Best regards,
>>
>> Michael
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181018/88ba2743/attachment-0001.html>