Hi Nirgil! An example would be: log { source(s_network); parser { json-parser(prefix(".json.")); }; destination { file("/tmp/output.log" template("$(format-json --key .json.* --exclude .json.age)")); }; }; Where the field "age" will be excluded, having the following input: <13>Sep 25 10:14:11 localhost some-app: {"name":"John Doe", "age":99, "foo":"dummy", "bar":"dummy", "baz":"dummy"} The output for that: {"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}} Please note, that the template() option overrides the entire format of the outgoing message, so you don't see the usual $HOST $DATE etc. fields. Therefore you need to re-create the original template: template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --exclude .json.age)\n") Example Output: <13>Sep 25 10:53:36 localhost some-app[444]: {"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}} The only exception to the above rule is the syslog() driver which sends the messages in RFC5424 format, and the template() option here only overrides the $MSG macro. Adding --shift-levels 2 removes the _json. key from the json template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --shift-levels 2 --exclude .json.age)\n") Output: <13>Sep 25 10:52:15 localhost some-app[444]: {"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"} Here is a description about format-json: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... And the options for format-json is describe under value-pairs section: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... Best Regards, Gabor Nirgil <nirgil@honeynet.cz> ezt írta (időpont: 2020. szept. 24., Cs, 10:10):
Hi Bazsi,
thanks you for informations.
Would you be so kind and provide me a sample, where to put $(format-json --exclude *payload*) ? In fact, I need to modify ${MSG} content only - the syslog header should be unchanged..
I have tried following, but this is not working for me..
template json_template {
template("${DATE} ${HOST} $PROGRAM: ${MSG}$(format-json --exclude *payload*)\n")
};
Or what --scope options should I use, experimented with various scope options, but without success.
Also tried --exclude * to see, if the option is removing anything, but none JSON fields are removed, it seems conversion is not working.
Thank you
Dne 24. 09. 20 v 7:44 Balazs Scheidler napsal(a):
Well, you could either remove the unnecessary field using a regexp but that's pretty fragile (due to escaping and stuff).
Or, you can reformat the message using the $(format-json) template function which can recreate the original json, but now with the "payload" field removed.
$(format-json) even has an --exclude option, meaning it can remove a field without having to explicitly unset it.
Bazsi
On Thu, Sep 24, 2020, 00:18 Nirgil <nirgil@honeynet.cz <mailto:nirgil@honeynet.cz>> wrote:
Hello all,
sorry to bother you, but I'm bit lost ;]
Trying to accomplish the following:
Have a JSON inside syslog message and I need to rewrite, or better to remove, one selected particular field, it contains lot of data and I just want to drop this one field before forwarding to different destination.
I have created a parser for JSON:
parser p_json {
channel { parser { json-parser (prefix(".json.")); }; }; };
Now I need remove or change a JSON field, lets say payload.
So I created the following rewrite rule and tried multiple options to remove or change the content
rewrite { unset(value("${.json.payload}")); unset(value(".json.payload")); groupunset(values("${.json.payload}")); set("TEST", value(".json.payload")); };
This is working on default fields ${HOST}, etc, but not for JSON fields.
Message can be reformatted via template ie:
template json_template {
template("${DATE} ${HOST} $PROGRAM: {\"payload\"\:\""${.json.payload}"\"}")
};
In that case, json.payload can be affected by appropriate rewrite rule.
But I'd like to affect the json.payload field inside the original ${MSG} without re-formatting whole message via custom template, because the original message has lot of optional fields, other inner JSON's and it is not so simple to made an universal template for this.
template json_template {
template("${DATE} ${HOST} $PROGRAM: ${MSG}")
};
Is there a way how to remove JSON field from original message (${MSG}) without reformatting the whole message via custom template ?
Thanks for any thoughts !
Sample of Message:
Sep 23 15:22:48 hostname myapplication: {"payload":"verylongpayload","field1":"value",.."lot of other fields, included inner JSON"...}
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq