Hi Gabor, thanks for the detailed description ! Kind regards Nirgil Dne 25. 09. 20 v 10:55 Nagy Gábor napsal(a):
Hi Nirgil!
An example would be:
log { source(s_network); parser { json-parser(prefix(".json.")); }; destination { file("/tmp/output.log" template("$(format-json --key .json.* --exclude .json.age)")); }; };
Where the field "age" will be excluded, having the following input: <13>Sep 25 10:14:11 localhost some-app: {"name":"John Doe", "age":99, "foo":"dummy", "bar":"dummy", "baz":"dummy"}
The output for that: {"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}
Please note, that the template() option overrides the entire format of the outgoing message, so you don't see the usual $HOST $DATE etc. fields. Therefore you need to re-create the original template: template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --exclude .json.age)\n") Example Output: <13>Sep 25 10:53:36 localhost some-app[444]: {"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}
The only exception to the above rule is the syslog() driver which sends the messages in RFC5424 format, and the template() option here only overrides the $MSG macro.
Adding --shift-levels 2 removes the _json. key from the json template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --shift-levels 2 --exclude .json.age)\n") Output: <13>Sep 25 10:52:15 localhost some-app[444]: {"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}
Here is a description about format-json: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... And the options for format-json is describe under value-pairs section: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
Best Regards, Gabor
Nirgil <nirgil@honeynet.cz <mailto:nirgil@honeynet.cz>> ezt írta (időpont: 2020. szept. 24., Cs, 10:10):
Hi Bazsi,
thanks you for informations.
Would you be so kind and provide me a sample, where to put $(format-json --exclude *payload*) ? In fact, I need to modify ${MSG} content only - the syslog header should be unchanged..
I have tried following, but this is not working for me..
template json_template {
template("${DATE} ${HOST} $PROGRAM: ${MSG}$(format-json --exclude *payload*)\n")
};
Or what --scope options should I use, experimented with various scope options, but without success.
Also tried --exclude * to see, if the option is removing anything, but none JSON fields are removed, it seems conversion is not working.
Thank you
Dne 24. 09. 20 v 7:44 Balazs Scheidler napsal(a): > Well, you could either remove the unnecessary field using a regexp but > that's pretty fragile (due to escaping and stuff). > > Or, you can reformat the message using the $(format-json) template > function which can recreate the original json, but now with the > "payload" field removed. > > $(format-json) even has an --exclude option, meaning it can remove a > field without having to explicitly unset it. > > Bazsi > > On Thu, Sep 24, 2020, 00:18 Nirgil <nirgil@honeynet.cz <mailto:nirgil@honeynet.cz> > <mailto:nirgil@honeynet.cz <mailto:nirgil@honeynet.cz>>> wrote: > > Hello all, > > sorry to bother you, but I'm bit lost ;] > > > > Trying to accomplish the following: > > Have a JSON inside syslog message and I need to rewrite, or better to > remove, one selected particular field, it contains lot of data and I > just want to drop this one field before forwarding to different > destination. > > I have created a parser for JSON: > > parser p_json { > > channel { > parser { > json-parser (prefix(".json.")); > }; > }; > }; > > > Now I need remove or change a JSON field, lets say payload. > > > So I created the following rewrite rule and tried multiple options to > remove or change the content > > rewrite { > unset(value("${.json.payload}")); > unset(value(".json.payload")); > groupunset(values("${.json.payload}")); > set("TEST", value(".json.payload")); > }; > > This is working on default fields ${HOST}, etc, but not for JSON fields. > > > Message can be reformatted via template ie: > > template json_template { > > template("${DATE} ${HOST} $PROGRAM: > {\"payload\"\:\""${.json.payload}"\"}") > > > }; > > In that case, json.payload can be affected by appropriate rewrite rule. > > But I'd like to affect the json.payload field inside the original ${MSG} > without re-formatting whole message via custom template, because the > original message has lot of optional fields, other inner JSON's and it > is not so simple to made an universal template for this. > > > > template json_template { > > template("${DATE} ${HOST} $PROGRAM: ${MSG}") > > }; > > > Is there a way how to remove JSON field from original message (${MSG}) > without reformatting the whole message via custom template ? > > > Thanks for any thoughts ! > > > Sample of Message: > > > Sep 23 15:22:48 hostname myapplication: > {"payload":"verylongpayload","field1":"value",.."lot of other fields, > included inner JSON"...} > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq