[syslog-ng] parsing JSON logs just won't work
Michael Niemand
michael.niemand at gmail.com
Thu Oct 18 10:30:25 UTC 2018
Thanks a lot for your answers. I realised I've got a super old version of
syslog-ng.
I upgraded to the most recent version and after some minor tweaks to the
config it works great now.
I'm sorry I didn't do this before writing to this list. Apologies!
Thanks again and best regards,
Michael
On Thu, Oct 18, 2018 at 10:35 AM Szemere, László <
laszlo.szemere at oneidentity.com> wrote:
> Dear Michael,
> thank you for the log message example! I putted it into a file called
> "message.json"
> I had to strip down your config (not tls, no `scl-root`, etc.). After
> that I started up a syslog-ng with this minimal config:
>
> @version: 3.5
> @include "scl.conf"
>
> template unitManagerTemplate {
> template("$(format-json --scope dot-nv-pairs) [sdid at 123456 X-OVH-TOKEN=\"
> XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
> };
>
> parser p_json {
> json-parser(prefix(".json."));
> };
>
> destination ovhPaaSLogs {
> tcp("127.0.0.1"
> port(6514),
> template(unitManagerTemplate),
> );
> };
>
> source s_net {
> file("/source/message.json"
> flags(no-parse)
> );
> };
>
> log {
> source(s_net);
> parser(p_json);
> destination(ovhPaaSLogs);
> };
>
>
>
> And I got the following output: (started syslog-ng with the following
> command: /install/sbin/syslog-ng *-Fdev* -f /source/syslog-ng.conf)
>
> [2018-10-18T08:27:49.450914] Incoming log entry;
> line='{"level":"error","message":"connection ended without disconnect
> receipt","timestamp":"2018-10-12T17:49:08.650Z"}'
> [2018-10-18T08:27:49.451015] LogSource window is empty;
> [2018-10-18T08:27:49.451076] Outgoing message;
> message='{"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection
> ended without disconnect receipt","level":"error"}} [sdid at 123456
> X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"\x0a'
>
> The destination in an another window:
>
> nc -kl 127.0.0.1 6514
> {"_json":{"timestamp":"2018-10-12T17:49:08.650Z","message":"connection
> ended without disconnect receipt","level":"error"}} [sdid at 123456
> X-OVH-TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXX"
>
>
> note: The paths (/install, /source, etc.) are coming from the fact I am
> using dbld (https://github.com/balabit/syslog-ng/tree/master/dbld), you
> might want to try it, to eliminate any environment effects. If it works,
> than we can compare your environment with dbld for differences.
>
> Br,
> Laci
>
>
>
> On Thu, Oct 18, 2018 at 8:18 AM, Nagy, Gábor <gabor.nagy at oneidentity.com>
> wrote:
>
>> Hi!
>>
>> For a quick idea, you can start syslog-ng in debug mode where you can see
>> details about the message parsing. You will see if the parsing or the
>> template had problems.
>>
>> You need to start syslog-ng with the -dv options to do that.
>>
>> Regards,
>> Gábor
>>
>> On Wed, 17 Oct 2018, 18:05 Michael Niemand, <michael.niemand at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I can’t get JSON parsing to work. I’ve consulted the documentation and
>>> Google but with no luck.
>>>
>>> I have an app, that puts out simple json log messages like:
>>>
>>> {"level":"error","message":"connection ended without disconnect
>>> receipt","timestamp":"2018-10-12T17:49:08.650Z"}
>>>
>>> All I want to do, is parse these 3 values and send them to a hosted
>>> Graylog cluster. Sending works, but the message gets inserted as
>>>
>>> application name: {"level"
>>> message: "error","message":"connection ended
>>> without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"}
>>>
>>> it's almost like syslog-ng doesn't even interpret the file as json. I
>>> tried different variants but I am at my wits end now...
>>>
>>> This is my config (on the application host; it should send the logs
>>> directly to the logging cluster)
>>>
>>> @version: 3.5
>>> @include "scl.conf"
>>> @include "`scl-root`/system/tty10.conf"
>>>
>>> options { chain_hostnames(off); flush_lines(0); use_dns(no);
>>> use_fqdn(no);
>>> owner("root"); group("adm"); perm(0640); stats_freq(0);
>>> bad_hostname("^gconfd$");
>>> };
>>>
>>> source s_src {
>>> file(
>>> “/var/log/worker/error.log"
>>> flags(no-parse)
>>> );
>>> };
>>>
>>> template unitManagerTemplate {
>>> template("$(format-json --scope dot-nv-pairs) [sdid at 123456
>>> X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n");
>>> };
>>>
>>> destination ovhPaaSLogs {
>>> tcp("gra2.logs.ovh.com"
>>> port(6514),
>>> template(unitManagerTemplate),
>>> ts_format("iso"),
>>> tls(peer-verify("require-trusted")
>>> ca_dir("/etc/ssl/certs/")),
>>> keep-alive(yes),
>>> so_keepalive(yes),
>>> );
>>> };
>>>
>>> parser p_json {
>>> json-parser(prefix(".json."));
>>> };
>>>
>>> log {
>>> source(s_src);
>>> parser(p_json);
>>> destination(ovhPaaSLogs);
>>> };
>>>
>>> @include "/etc/syslog-ng/conf.d/"
>>>
>>>
>>> I also tried a different a template variant like this:
>>>
>>> template("${.json.level} ${.json.message} ${.json.timestamp}
>>> [sdid at 123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”);
>>>
>>> I also tried parsing the messages as text:
>>>
>>>
>>> template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"}
>>> - [sdid at 32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\"
>>> facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n");
>>>
>>> What shows up in Graylog is absolutely identical (like described in the
>>> beginning). In fact, every variant that I tried changed absolutely nothing.
>>> The conf.d folder is empty though.
>>> I’d appreciate any help!
>>>
>>>
>>> Best regards,
>>>
>>> Michael
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
--
LargePrefPlaceholder-XKUz1MEJBwkOM
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181018/e3165ad3/attachment-0001.html>
More information about the syslog-ng
mailing list