[syslog-ng] Rewrite JSON field without re-formating via custom template

[Nirgil]

Hi Bazsi,

thanks you for informations.

Would you be so kind and provide me a sample, where to put $(format-json
--exclude *payload*) ? In fact, I need to modify ${MSG} content only -
the syslog header should be unchanged..

I have tried following, but this is not working for me..

template json_template {

             template("${DATE} ${HOST} $PROGRAM: ${MSG}$(format-json
--exclude *payload*)\n")

};

Or what --scope options should I use, experimented with various scope
options, but without success.

Also tried --exclude * to see, if the option is removing anything, but
none JSON fields are removed, it seems conversion is not working.

Thank you







Dne 24. 09. 20 v 7:44 Balazs Scheidler napsal(a):
> Well, you could either remove the unnecessary field using a regexp but
> that's pretty fragile (due to escaping and stuff).
> 
> Or, you can reformat the message using the $(format-json) template
> function which can recreate the original json, but now with the
> "payload" field removed.
> 
> $(format-json) even has an --exclude option, meaning it can remove a
> field without having to explicitly unset it.
> 
> Bazsi
> 
> On Thu, Sep 24, 2020, 00:18 Nirgil <nirgil at honeynet.cz
> <mailto:nirgil at honeynet.cz>> wrote:
> 
>     Hello all,
> 
>     sorry to bother you, but I'm bit lost ;]
> 
> 
> 
>     Trying to accomplish the following:
> 
>     Have a JSON inside syslog message and I need to rewrite, or better to
>     remove, one selected particular field, it contains lot of data and I
>     just want to drop this one field before forwarding to different
>     destination.
> 
>     I have created a parser for JSON:
> 
>     parser p_json {
> 
>             channel {
>                     parser {
>                             json-parser (prefix(".json."));
>                     };
>             };
>     };
> 
> 
>     Now I need remove or change a JSON field, lets say payload.
> 
> 
>     So I created the following rewrite rule and tried multiple options to
>     remove or change the content
> 
>     rewrite {
>             unset(value("${.json.payload}"));
>             unset(value(".json.payload"));
>             groupunset(values("${.json.payload}"));
>             set("TEST", value(".json.payload"));
>     };
> 
>     This is working on default fields ${HOST}, etc, but not for JSON fields.
> 
> 
>     Message can be reformatted via template ie:
> 
>     template json_template {
> 
>             template("${DATE} ${HOST} $PROGRAM:
>     {\"payload\"\:\""${.json.payload}"\"}")
> 
> 
>     };
> 
>     In that case, json.payload can be affected by appropriate rewrite rule.
> 
>     But I'd like to affect the json.payload field inside the original ${MSG}
>     without re-formatting whole message via custom template, because the
>     original message has lot of optional fields, other inner JSON's and it
>     is not so simple to made an universal template for this.
> 
> 
> 
>     template json_template {
> 
>             template("${DATE} ${HOST} $PROGRAM: ${MSG}")
> 
>     };
> 
> 
>     Is there a way how to remove JSON field from original message (${MSG})
>     without reformatting the whole message via custom template ?
> 
> 
>     Thanks for any thoughts !
> 
> 
>     Sample of Message:
> 
> 
>     Sep 23 15:22:48 hostname myapplication:
>     {"payload":"verylongpayload","field1":"value",.."lot of other fields,
>     included inner JSON"...}
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>