[syslog-ng] Rewrite JSON field without re-formating via custom template
Nirgil
nirgil at honeynet.cz
Tue Sep 29 13:49:10 UTC 2020
Hi Gabor,
thanks for the detailed description !
Kind regards
Nirgil
Dne 25. 09. 20 v 10:55 Nagy Gábor napsal(a):
> Hi Nirgil!
>
> An example would be:
>
> log {
> source(s_network);
> parser { json-parser(prefix(".json.")); };
> destination { file("/tmp/output.log" template("$(format-json --key
> .json.* --exclude .json.age)")); };
> };
>
> Where the field "age" will be excluded, having the following input:
> <13>Sep 25 10:14:11 localhost some-app: {"name":"John Doe", "age":99,
> "foo":"dummy", "bar":"dummy", "baz":"dummy"}
>
> The output for that:
> {"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}
>
> Please note, that the template() option overrides the entire format of
> the outgoing message, so you don't see the usual $HOST $DATE etc. fields.
> Therefore you need to re-create the original template:
> template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.*
> --exclude .json.age)\n")
> Example Output:
> <13>Sep 25 10:53:36 localhost some-app[444]: {"_json":{"name":"John
> Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}
>
> The only exception to the above rule is the syslog() driver which sends
> the messages in RFC5424 format, and the template() option here only
> overrides the $MSG macro.
>
> Adding --shift-levels 2 removes the _json. key from the json
> template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.*
> --shift-levels 2 --exclude .json.age)\n")
> Output:
> <13>Sep 25 10:52:15 localhost some-app[444]: {"name":"John
> Doe","foo":"dummy","baz":"dummy","bar":"dummy"}
>
> Here is a description about format-json:
> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/65#template-function-format-json
> And the options for format-json is describe under value-pairs section:
> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/9#TOPIC-1430930
>
> Best Regards,
> Gabor
>
> Nirgil <nirgil at honeynet.cz <mailto:nirgil at honeynet.cz>> ezt írta
> (időpont: 2020. szept. 24., Cs, 10:10):
>
> Hi Bazsi,
>
> thanks you for informations.
>
> Would you be so kind and provide me a sample, where to put $(format-json
> --exclude *payload*) ? In fact, I need to modify ${MSG} content only -
> the syslog header should be unchanged..
>
> I have tried following, but this is not working for me..
>
> template json_template {
>
> template("${DATE} ${HOST} $PROGRAM: ${MSG}$(format-json
> --exclude *payload*)\n")
>
> };
>
> Or what --scope options should I use, experimented with various scope
> options, but without success.
>
> Also tried --exclude * to see, if the option is removing anything, but
> none JSON fields are removed, it seems conversion is not working.
>
> Thank you
>
>
>
>
>
>
>
> Dne 24. 09. 20 v 7:44 Balazs Scheidler napsal(a):
> > Well, you could either remove the unnecessary field using a regexp but
> > that's pretty fragile (due to escaping and stuff).
> >
> > Or, you can reformat the message using the $(format-json) template
> > function which can recreate the original json, but now with the
> > "payload" field removed.
> >
> > $(format-json) even has an --exclude option, meaning it can remove a
> > field without having to explicitly unset it.
> >
> > Bazsi
> >
> > On Thu, Sep 24, 2020, 00:18 Nirgil <nirgil at honeynet.cz
> <mailto:nirgil at honeynet.cz>
> > <mailto:nirgil at honeynet.cz <mailto:nirgil at honeynet.cz>>> wrote:
> >
> > Hello all,
> >
> > sorry to bother you, but I'm bit lost ;]
> >
> >
> >
> > Trying to accomplish the following:
> >
> > Have a JSON inside syslog message and I need to rewrite, or
> better to
> > remove, one selected particular field, it contains lot of data
> and I
> > just want to drop this one field before forwarding to different
> > destination.
> >
> > I have created a parser for JSON:
> >
> > parser p_json {
> >
> > channel {
> > parser {
> > json-parser (prefix(".json."));
> > };
> > };
> > };
> >
> >
> > Now I need remove or change a JSON field, lets say payload.
> >
> >
> > So I created the following rewrite rule and tried multiple
> options to
> > remove or change the content
> >
> > rewrite {
> > unset(value("${.json.payload}"));
> > unset(value(".json.payload"));
> > groupunset(values("${.json.payload}"));
> > set("TEST", value(".json.payload"));
> > };
> >
> > This is working on default fields ${HOST}, etc, but not for
> JSON fields.
> >
> >
> > Message can be reformatted via template ie:
> >
> > template json_template {
> >
> > template("${DATE} ${HOST} $PROGRAM:
> > {\"payload\"\:\""${.json.payload}"\"}")
> >
> >
> > };
> >
> > In that case, json.payload can be affected by appropriate
> rewrite rule.
> >
> > But I'd like to affect the json.payload field inside the
> original ${MSG}
> > without re-formatting whole message via custom template,
> because the
> > original message has lot of optional fields, other inner
> JSON's and it
> > is not so simple to made an universal template for this.
> >
> >
> >
> > template json_template {
> >
> > template("${DATE} ${HOST} $PROGRAM: ${MSG}")
> >
> > };
> >
> >
> > Is there a way how to remove JSON field from original message
> (${MSG})
> > without reformatting the whole message via custom template ?
> >
> >
> > Thanks for any thoughts !
> >
> >
> > Sample of Message:
> >
> >
> > Sep 23 15:22:48 hostname myapplication:
> > {"payload":"verylongpayload","field1":"value",.."lot of other
> fields,
> > included inner JSON"...}
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
More information about the syslog-ng
mailing list