[syslog-ng] Rewrite JSON field without re-formating via custom template
Nagy Gábor
gabor.hl at gmail.com
Fri Sep 25 08:55:59 UTC 2020
Hi Nirgil!
An example would be:
log {
source(s_network);
parser { json-parser(prefix(".json.")); };
destination { file("/tmp/output.log" template("$(format-json --key
.json.* --exclude .json.age)")); };
};
Where the field "age" will be excluded, having the following input:
<13>Sep 25 10:14:11 localhost some-app: {"name":"John Doe", "age":99,
"foo":"dummy", "bar":"dummy", "baz":"dummy"}
The output for that:
{"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}
Please note, that the template() option overrides the entire format of the
outgoing message, so you don't see the usual $HOST $DATE etc. fields.
Therefore you need to re-create the original template:
template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.*
--exclude .json.age)\n")
Example Output:
<13>Sep 25 10:53:36 localhost some-app[444]: {"_json":{"name":"John
Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}
The only exception to the above rule is the syslog() driver which sends the
messages in RFC5424 format, and the template() option here only overrides
the $MSG macro.
Adding --shift-levels 2 removes the _json. key from the json
template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.*
--shift-levels 2 --exclude .json.age)\n")
Output:
<13>Sep 25 10:52:15 localhost some-app[444]: {"name":"John
Doe","foo":"dummy","baz":"dummy","bar":"dummy"}
Here is a description about format-json:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/65#template-function-format-json
And the options for format-json is describe under value-pairs section:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/9#TOPIC-1430930
Best Regards,
Gabor
Nirgil <nirgil at honeynet.cz> ezt írta (időpont: 2020. szept. 24., Cs, 10:10):
> Hi Bazsi,
>
> thanks you for informations.
>
> Would you be so kind and provide me a sample, where to put $(format-json
> --exclude *payload*) ? In fact, I need to modify ${MSG} content only -
> the syslog header should be unchanged..
>
> I have tried following, but this is not working for me..
>
> template json_template {
>
> template("${DATE} ${HOST} $PROGRAM: ${MSG}$(format-json
> --exclude *payload*)\n")
>
> };
>
> Or what --scope options should I use, experimented with various scope
> options, but without success.
>
> Also tried --exclude * to see, if the option is removing anything, but
> none JSON fields are removed, it seems conversion is not working.
>
> Thank you
>
>
>
>
>
>
>
> Dne 24. 09. 20 v 7:44 Balazs Scheidler napsal(a):
> > Well, you could either remove the unnecessary field using a regexp but
> > that's pretty fragile (due to escaping and stuff).
> >
> > Or, you can reformat the message using the $(format-json) template
> > function which can recreate the original json, but now with the
> > "payload" field removed.
> >
> > $(format-json) even has an --exclude option, meaning it can remove a
> > field without having to explicitly unset it.
> >
> > Bazsi
> >
> > On Thu, Sep 24, 2020, 00:18 Nirgil <nirgil at honeynet.cz
> > <mailto:nirgil at honeynet.cz>> wrote:
> >
> > Hello all,
> >
> > sorry to bother you, but I'm bit lost ;]
> >
> >
> >
> > Trying to accomplish the following:
> >
> > Have a JSON inside syslog message and I need to rewrite, or better to
> > remove, one selected particular field, it contains lot of data and I
> > just want to drop this one field before forwarding to different
> > destination.
> >
> > I have created a parser for JSON:
> >
> > parser p_json {
> >
> > channel {
> > parser {
> > json-parser (prefix(".json."));
> > };
> > };
> > };
> >
> >
> > Now I need remove or change a JSON field, lets say payload.
> >
> >
> > So I created the following rewrite rule and tried multiple options to
> > remove or change the content
> >
> > rewrite {
> > unset(value("${.json.payload}"));
> > unset(value(".json.payload"));
> > groupunset(values("${.json.payload}"));
> > set("TEST", value(".json.payload"));
> > };
> >
> > This is working on default fields ${HOST}, etc, but not for JSON
> fields.
> >
> >
> > Message can be reformatted via template ie:
> >
> > template json_template {
> >
> > template("${DATE} ${HOST} $PROGRAM:
> > {\"payload\"\:\""${.json.payload}"\"}")
> >
> >
> > };
> >
> > In that case, json.payload can be affected by appropriate rewrite
> rule.
> >
> > But I'd like to affect the json.payload field inside the original
> ${MSG}
> > without re-formatting whole message via custom template, because the
> > original message has lot of optional fields, other inner JSON's and
> it
> > is not so simple to made an universal template for this.
> >
> >
> >
> > template json_template {
> >
> > template("${DATE} ${HOST} $PROGRAM: ${MSG}")
> >
> > };
> >
> >
> > Is there a way how to remove JSON field from original message
> (${MSG})
> > without reformatting the whole message via custom template ?
> >
> >
> > Thanks for any thoughts !
> >
> >
> > Sample of Message:
> >
> >
> > Sep 23 15:22:48 hostname myapplication:
> > {"payload":"verylongpayload","field1":"value",.."lot of other fields,
> > included inner JSON"...}
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200925/eeb8a466/attachment.html>
More information about the syslog-ng
mailing list