[syslog-ng] Rewrite JSON field without re-formating via custom template

[Balazs Scheidler]

Well, you could either remove the unnecessary field using a regexp but
that's pretty fragile (due to escaping and stuff).

Or, you can reformat the message using the $(format-json) template function
which can recreate the original json, but now with the "payload" field
removed.

$(format-json) even has an --exclude option, meaning it can remove a field
without having to explicitly unset it.

Bazsi

On Thu, Sep 24, 2020, 00:18 Nirgil <nirgil at honeynet.cz> wrote:

> Hello all,
>
> sorry to bother you, but I'm bit lost ;]
>
>
>
> Trying to accomplish the following:
>
> Have a JSON inside syslog message and I need to rewrite, or better to
> remove, one selected particular field, it contains lot of data and I
> just want to drop this one field before forwarding to different
> destination.
>
> I have created a parser for JSON:
>
> parser p_json {
>
>         channel {
>                 parser {
>                         json-parser (prefix(".json."));
>                 };
>         };
> };
>
>
> Now I need remove or change a JSON field, lets say payload.
>
>
> So I created the following rewrite rule and tried multiple options to
> remove or change the content
>
> rewrite {
>         unset(value("${.json.payload}"));
>         unset(value(".json.payload"));
>         groupunset(values("${.json.payload}"));
>         set("TEST", value(".json.payload"));
> };
>
> This is working on default fields ${HOST}, etc, but not for JSON fields.
>
>
> Message can be reformatted via template ie:
>
> template json_template {
>
>         template("${DATE} ${HOST} $PROGRAM:
> {\"payload\"\:\""${.json.payload}"\"}")
>
>
> };
>
> In that case, json.payload can be affected by appropriate rewrite rule.
>
> But I'd like to affect the json.payload field inside the original ${MSG}
> without re-formatting whole message via custom template, because the
> original message has lot of optional fields, other inner JSON's and it
> is not so simple to made an universal template for this.
>
>
>
> template json_template {
>
>         template("${DATE} ${HOST} $PROGRAM: ${MSG}")
>
> };
>
>
> Is there a way how to remove JSON field from original message (${MSG})
> without reformatting the whole message via custom template ?
>
>
> Thanks for any thoughts !
>
>
> Sample of Message:
>
>
> Sep 23 15:22:48 hostname myapplication:
> {"payload":"verylongpayload","field1":"value",.."lot of other fields,
> included inner JSON"...}
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200924/b0380b57/attachment.html>