<div dir="ltr">Hi Nirgil!<br><div><br></div><div>An example would be:</div><div><br></div><div>log {</div><div> source(s_network);<br></div><div> parser { json-parser(prefix(".json.")); };</div><div> destination { file("/tmp/output.log" template("$(format-json --key .json.* --exclude .json.age)")); }; <br></div><div>};<br></div><div><br></div><div>Where the field "age" will be excluded, having the following input:<br></div><div><13>Sep 25 10:14:11 localhost some-app: {"name":"John Doe", "age":99, "foo":"dummy", "bar":"dummy", "baz":"dummy"}</div><div><br>The output for that:<br>{"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}<br></div><div><br></div><div>Please note, that the template() option overrides the entire format of the outgoing message, so you don't see the usual $HOST $DATE etc. fields.<br></div><div></div><div>Therefore you need to re-create the original template:</div><div>template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --exclude .json.age)\n")</div><div>Example Output:<br></div><div><13>Sep 25 10:53:36 localhost some-app[444]: {"_json":{"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}}</div><div><br></div><div>The only exception to the above rule is the syslog() driver which sends the messages in RFC5424 format, and the template() option here only overrides the $MSG macro.</div><div> </div><div></div><div><br></div><div>Adding --shift-levels 2 removes the _json. key from the json</div><div>template("<${PRI}>${DATE} ${HOST} ${MSGHDR}$(format-json --key .json.* --shift-levels 2 --exclude .json.age)\n")</div><div>Output:<br></div><div><13>Sep 25 10:52:15 localhost some-app[444]: {"name":"John Doe","foo":"dummy","baz":"dummy","bar":"dummy"}<br></div><div><br></div><div>Here is a description about format-json:<br><a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/65#template-function-format-json">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/65#template-function-format-json</a></div><div>And the options for format-json is describe under value-pairs section:<br><a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/9#TOPIC-1430930">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/9#TOPIC-1430930</a></div><br><div>Best Regards,<br></div><div>Gabor<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Nirgil <<a href="mailto:nirgil@honeynet.cz">nirgil@honeynet.cz</a>> ezt írta (időpont: 2020. szept. 24., Cs, 10:10):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Bazsi,<br>
<br>
thanks you for informations.<br>
<br>
Would you be so kind and provide me a sample, where to put $(format-json<br>
--exclude *payload*) ? In fact, I need to modify ${MSG} content only -<br>
the syslog header should be unchanged..<br>
<br>
I have tried following, but this is not working for me..<br>
<br>
template json_template {<br>
<br>
template("${DATE} ${HOST} $PROGRAM: ${MSG}$(format-json<br>
--exclude *payload*)\n")<br>
<br>
};<br>
<br>
Or what --scope options should I use, experimented with various scope<br>
options, but without success.<br>
<br>
Also tried --exclude * to see, if the option is removing anything, but<br>
none JSON fields are removed, it seems conversion is not working.<br>
<br>
Thank you<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Dne 24. 09. 20 v 7:44 Balazs Scheidler napsal(a):<br>
> Well, you could either remove the unnecessary field using a regexp but<br>
> that's pretty fragile (due to escaping and stuff).<br>
> <br>
> Or, you can reformat the message using the $(format-json) template<br>
> function which can recreate the original json, but now with the<br>
> "payload" field removed.<br>
> <br>
> $(format-json) even has an --exclude option, meaning it can remove a<br>
> field without having to explicitly unset it.<br>
> <br>
> Bazsi<br>
> <br>
> On Thu, Sep 24, 2020, 00:18 Nirgil <<a href="mailto:nirgil@honeynet.cz" target="_blank">nirgil@honeynet.cz</a><br>
> <mailto:<a href="mailto:nirgil@honeynet.cz" target="_blank">nirgil@honeynet.cz</a>>> wrote:<br>
> <br>
> Hello all,<br>
> <br>
> sorry to bother you, but I'm bit lost ;]<br>
> <br>
> <br>
> <br>
> Trying to accomplish the following:<br>
> <br>
> Have a JSON inside syslog message and I need to rewrite, or better to<br>
> remove, one selected particular field, it contains lot of data and I<br>
> just want to drop this one field before forwarding to different<br>
> destination.<br>
> <br>
> I have created a parser for JSON:<br>
> <br>
> parser p_json {<br>
> <br>
> channel {<br>
> parser {<br>
> json-parser (prefix(".json."));<br>
> };<br>
> };<br>
> };<br>
> <br>
> <br>
> Now I need remove or change a JSON field, lets say payload.<br>
> <br>
> <br>
> So I created the following rewrite rule and tried multiple options to<br>
> remove or change the content<br>
> <br>
> rewrite {<br>
> unset(value("${.json.payload}"));<br>
> unset(value(".json.payload"));<br>
> groupunset(values("${.json.payload}"));<br>
> set("TEST", value(".json.payload"));<br>
> };<br>
> <br>
> This is working on default fields ${HOST}, etc, but not for JSON fields.<br>
> <br>
> <br>
> Message can be reformatted via template ie:<br>
> <br>
> template json_template {<br>
> <br>
> template("${DATE} ${HOST} $PROGRAM:<br>
> {\"payload\"\:\""${.json.payload}"\"}")<br>
> <br>
> <br>
> };<br>
> <br>
> In that case, json.payload can be affected by appropriate rewrite rule.<br>
> <br>
> But I'd like to affect the json.payload field inside the original ${MSG}<br>
> without re-formatting whole message via custom template, because the<br>
> original message has lot of optional fields, other inner JSON's and it<br>
> is not so simple to made an universal template for this.<br>
> <br>
> <br>
> <br>
> template json_template {<br>
> <br>
> template("${DATE} ${HOST} $PROGRAM: ${MSG}")<br>
> <br>
> };<br>
> <br>
> <br>
> Is there a way how to remove JSON field from original message (${MSG})<br>
> without reformatting the whole message via custom template ?<br>
> <br>
> <br>
> Thanks for any thoughts !<br>
> <br>
> <br>
> Sample of Message:<br>
> <br>
> <br>
> Sep 23 15:22:48 hostname myapplication:<br>
> {"payload":"verylongpayload","field1":"value",.."lot of other fields,<br>
> included inner JSON"...}<br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation:<br>
> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
> <br>
> <br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
> <br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>