Hi Nirgil!

An example would be:

log {

source(s_network);

parser { json-parser(prefix(".json.")); };

destination { file("/tmp/output.log" template("$(format-json --key .json.* --exclude .json.age)")); };

};

Where the field "age" will be excluded, having the following input:

<13>Sep 25 10:14:11 localhost some-app: {"name":"John Doe", "age":99, "foo":"dummy", "bar":"dummy", "baz":"dummy"}

The output for that:
{"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}

Please note, that the template() option overrides the entire format of the outgoing message, so you don't see the usual $HOST $DATE etc. fields.

Therefore you need to re-create the original template:

template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --exclude .json.age)\n")

Example Output:

<13>Sep 25 10:53:36 localhost some-app[444]: {"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}

The only exception to the above rule is the syslog() driver which sends the messages in RFC5424 format, and the template() option here only overrides the $MSG macro.

Adding --shift-levels 2 removes the _json. key from the json

template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --shift-levels 2 --exclude .json.age)\n")

Output:

<13>Sep 25 10:52:15 localhost some-app[444]: {"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}

Here is a description about format-json:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/65#template-function-format-json

And the options for format-json is describe under value-pairs section:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/9#TOPIC-1430930

Best Regards,

Gabor

Nirgil <nirgil@honeynet.cz> ezt írta (időpont: 2020. szept. 24., Cs, 10:10):

Hi Bazsi,

thanks you for informations.

Would you be so kind and provide me a sample, where to put $(format-json
--exclude *payload*) ? In fact, I need to modify ${MSG} content only -
the syslog header should be unchanged..

I have tried following, but this is not working for me..

template json_template {

template("${DATE} ${HOST} $PROGRAM: ${MSG}$(format-json
--exclude *payload*)\n")

};

Or what --scope options should I use, experimented with various scope
options, but without success.

Also tried --exclude * to see, if the option is removing anything, but
none JSON fields are removed, it seems conversion is not working.

Thank you

Dne 24. 09. 20 v 7:44 Balazs Scheidler napsal(a):
> Well, you could either remove the unnecessary field using a regexp but
> that's pretty fragile (due to escaping and stuff).
>
> Or, you can reformat the message using the $(format-json) template
> function which can recreate the original json, but now with the
> "payload" field removed.
>
> $(format-json) even has an --exclude option, meaning it can remove a
> field without having to explicitly unset it.
>
> Bazsi
>
> On Thu, Sep 24, 2020, 00:18 Nirgil <nirgil@honeynet.cz
> <mailto:nirgil@honeynet.cz>> wrote:
>
> Hello all,
>
> sorry to bother you, but I'm bit lost ;]
>
>
>
> Trying to accomplish the following:
>
> Have a JSON inside syslog message and I need to rewrite, or better to
> remove, one selected particular field, it contains lot of data and I
> just want to drop this one field before forwarding to different
> destination.
>
> I have created a parser for JSON:
>
> parser p_json {
>
>    channel {
>    parser {
>    json-parser (prefix(".json."));
>    };
>    };
> };
>
>
> Now I need remove or change a JSON field, lets say payload.
>
>
> So I created the following rewrite rule and tried multiple options to
> remove or change the content
>
> rewrite {
>    unset(value("${.json.payload}"));
>    unset(value(".json.payload"));
>    groupunset(values("${.json.payload}"));
>    set("TEST", value(".json.payload"));
> };
>
> This is working on default fields ${HOST}, etc, but not for JSON fields.
>
>
> Message can be reformatted via template ie:
>
> template json_template {
>
>    template("${DATE} ${HOST} $PROGRAM:
> {\"payload\"\:\""${.json.payload}"\"}")
>
>
> };
>
> In that case, json.payload can be affected by appropriate rewrite rule.
>
> But I'd like to affect the json.payload field inside the original ${MSG}
> without re-formatting whole message via custom template, because the
> original message has lot of optional fields, other inner JSON's and it
> is not so simple to made an universal template for this.
>
>
>
> template json_template {
>
>    template("${DATE} ${HOST} $PROGRAM: ${MSG}")
>
> };
>
>
> Is there a way how to remove JSON field from original message (${MSG})
> without reformatting the whole message via custom template ?
>
>
> Thanks for any thoughts !
>
>
> Sample of Message:
>
>
> Sep 23 15:22:48 hostname myapplication:
> {"payload":"verylongpayload","field1":"value",.."lot of other fields,
> included inner JSON"...}
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq