<div dir="ltr"><div dir="ltr">Hi <span class="gmail-qu" role="gridcell" tabindex="-1"><span name="Александр Масленников" class="gmail-gD"><span>Александр</span></span></span>,<br></div><div><br></div><div>The problem is that the "answers" element of this entry is a list of dicts. Previously lists were not even properly supported. Recent versions of syslog-ng (that is very recent we merged quite a few PRs in this area, e.g. <a href="https://github.com/syslog-ng/syslog-ng/pull/3885">https://github.com/syslog-ng/syslog-ng/pull/3885</a>) do support lists embedded in JSON objects, and turn them into a list of strings (in case they have simple types) or keep them a literal "JSON" typed value, which is reproduced verbatim.</div><div><br></div><div>I have configuration:</div><div><br></div><div>@version: 3.37<br><br>log { <br> source { tcp(port(2000) flags(no-parse)); };<br> parser { json-parser(prefix(".json.")); };<br> destination { file("/tmp/json.out" template("$(format-flat-json --subkeys .json.)\n")); };<br>};<br></div><br><div>With a recent version of syslog-ng, the $(format-flat-json) would look like this:</div><div><br></div><div>{<br> "timestamp-rfc3339": "2022-06-06T08:47:58.797332215Z",<br> "response-port": "53",<br> "response-ip": "192.168.yy.zz",<br> "rcode": "NOERROR",<br> "query-port": "51000",<br> "query-ip": "192.168.xx.zz",<br> "qtype": "TXT",<br> "qname": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "protocol": "TCP",<br> "operation": "CLIENT_RESPONSE",<br> "length": "691",<br> "latency": "0.000000",<br> "identity": "<a href="http://ns-server.example.com">ns-server.example.com</a>",<br> "family": "INET",<br> "country-isocode": "-",<br> "answers": "[{\"name\":\"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>\",\"rdatatype\":\"TXT\",\"ttl\":600,\"rdata\":\"dnsaddr=\\/dnsaddr\\/<a href="http://ams-2.bootstrap.libp2p.io">ams-2.bootstrap.libp2p.io</a>\\/p2p\\/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb\"},{\"name\":\"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>\",\"rdatatype\":\"TXT\",\"ttl\":600,\"rdata\":\"dnsaddr=\\/dnsaddr\\/<a href="http://sjc-1.bootstrap.libp2p.io">sjc-1.bootstrap.libp2p.io</a>\\/p2p\\/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN\"},{\"name\":\"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>\",\"rdatatype\":\"TXT\",\"ttl\":600,\"rdata\":\"dnsaddr=\\/dnsaddr\\/<a href="http://ewr-1.bootstrap.libp2p.io">ewr-1.bootstrap.libp2p.io</a>\\/p2p\\/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa\"},{\"name\":\"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>\",\"rdatatype\":\"TXT\",\"ttl\":600,\"rdata\":\"dnsaddr=\\/dnsaddr\\/<a href="http://ams-rust.bootstrap.libp2p.io">ams-rust.bootstrap.libp2p.io</a>\\/p2p\\/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS\"},{\"name\":\"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>\",\"rdatatype\":\"TXT\",\"ttl\":600,\"rdata\":\"dnsaddr=\\/dnsaddr\\/<a href="http://sjc-2.bootstrap.libp2p.io">sjc-2.bootstrap.libp2p.io</a>\\/p2p\\/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp\"},{\"name\":\"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>\",\"rdatatype\":\"TXT\",\"ttl\":600,\"rdata\":\"dnsaddr=\\/dnsaddr\\/<a href="http://nrt-1.bootstrap.libp2p.io">nrt-1.bootstrap.libp2p.io</a>\\/p2p\\/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt\"}]"<br>}</div><div><br></div><div>"anwers" is now a string, containing a JSON expression that contains a list of objects. <br></div><div><br></div><div>If I set the syslog-ng config version to 4.0, I get the new typing behaviour, which means that ${.json.answers} is now a JSON literal, so the output becomes this:</div><div><br></div><div>{<br> "timestamp-rfc3339": "2022-06-06T08:47:58.797332215Z",<br> "response-port": "53",<br> "response-ip": "192.168.yy.zz",<br> "rcode": "NOERROR",<br> "query-port": "51000",<br> "query-ip": "192.168.xx.zz",<br> "qtype": "TXT",<br> "qname": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "protocol": "TCP",<br> "operation": "CLIENT_RESPONSE",<br> "length": 691,<br> "latency": "0.000000",<br> "identity": "<a href="http://ns-server.example.com">ns-server.example.com</a>",<br> "family": "INET",<br> "country-isocode": "-",<br> "answers": [<br> {<br> "name": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "rdatatype": "TXT",<br> "ttl": 600,<br> "rdata": "dnsaddr=/dnsaddr/<a href="http://ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb">ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb</a>"<br> },<br> {<br> "name": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "rdatatype": "TXT",<br> "ttl": 600,<br> "rdata": "dnsaddr=/dnsaddr/<a href="http://sjc-1.bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN">sjc-1.bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN</a>"<br> },<br> {<br> "name": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "rdatatype": "TXT",<br> "ttl": 600,<br> "rdata": "dnsaddr=/dnsaddr/<a href="http://ewr-1.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa">ewr-1.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa</a>"<br> },<br> {<br> "name": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "rdatatype": "TXT",<br> "ttl": 600,<br> "rdata": "dnsaddr=/dnsaddr/<a href="http://ams-rust.bootstrap.libp2p.io/p2p/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS">ams-rust.bootstrap.libp2p.io/p2p/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS</a>"<br> },<br> {<br> "name": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "rdatatype": "TXT",<br> "ttl": 600,<br> "rdata": "dnsaddr=/dnsaddr/<a href="http://sjc-2.bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp">sjc-2.bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp</a>"<br> },<br> {<br> "name": "_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>",<br> "rdatatype": "TXT",<br> "ttl": 600,<br> "rdata": "dnsaddr=/dnsaddr/<a href="http://nrt-1.bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt">nrt-1.bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt</a>"<br> }<br> ]<br>}<br></div><div><br></div><div>e.g. at least "answers" becomes an array and not a string. That's only slightly better. Let me parse the embedded list a second time. I am adding this parser to the config:</div><br><div>@version: 4.0<br><br>log { <br> source { tcp(port(2000) flags(no-parse)); };<br><br> parser { json-parser(prefix(".json.")); };<br><br> parser { json-parser(prefix(".json.answers") template("${.json.answers}")); };<br><br> destination { file("/tmp/json.out" template("$(format-flat-json --subkeys .json.)\n")); };<br>};<br></div><div><br></div><div>The 2nd parser finds that the input to be parsed is a list. The new syslog-ng 4.0 behaviour is to parse elements into $1, $2, etc. This is the trace output of the 2nd JSON parser:</div><div><br></div><div>[2022-06-10T18:20:12.927168] Setting value; name='1', value='{"name":"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>","rdatatype":"TXT","ttl":600,"rdata":"dnsaddr=\/dnsaddr\/<a href="http://ams-2.bootstrap.libp2p.io">ams-2.bootstrap.libp2p.io</a>\/p2p\/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb"}', type='json', msg='0x7ffff0014190', rcptid='102'<br>[2022-06-10T18:20:12.927179] Setting value; name='2', value='{"name":"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>","rdatatype":"TXT","ttl":600,"rdata":"dnsaddr=\/dnsaddr\/<a href="http://sjc-1.bootstrap.libp2p.io">sjc-1.bootstrap.libp2p.io</a>\/p2p\/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN"}', type='json', msg='0x7ffff0014190', rcptid='102'<br>[2022-06-10T18:20:12.927190] Setting value; name='3', value='{"name":"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>","rdatatype":"TXT","ttl":600,"rdata":"dnsaddr=\/dnsaddr\/<a href="http://ewr-1.bootstrap.libp2p.io">ewr-1.bootstrap.libp2p.io</a>\/p2p\/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"}', type='json', msg='0x7ffff0014190', rcptid='102'<br>[2022-06-10T18:20:12.927199] Setting value; name='4', value='{"name":"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>","rdatatype":"TXT","ttl":600,"rdata":"dnsaddr=\/dnsaddr\/<a href="http://ams-rust.bootstrap.libp2p.io">ams-rust.bootstrap.libp2p.io</a>\/p2p\/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS"}', type='json', msg='0x7ffff0014190', rcptid='102'<br>[2022-06-10T18:20:12.927207] Setting value; name='5', value='{"name":"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>","rdatatype":"TXT","ttl":600,"rdata":"dnsaddr=\/dnsaddr\/<a href="http://sjc-2.bootstrap.libp2p.io">sjc-2.bootstrap.libp2p.io</a>\/p2p\/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp"}', type='json', msg='0x7ffff0014190', rcptid='102'<br>[2022-06-10T18:20:12.927216] Setting value; name='6', value='{"name":"_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>","rdatatype":"TXT","ttl":600,"rdata":"dnsaddr=\/dnsaddr\/<a href="http://nrt-1.bootstrap.libp2p.io">nrt-1.bootstrap.libp2p.io</a>\/p2p\/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"}', type='json', msg='0x7ffff0014190', rcptid='102'</div><div><br></div><div>e.g. $1 would be the first element of the JSON array, $2 being the 2nd and so on. We can turn these matches into a syslog-ng list, using the special macro "$*" and manipulate it using the list related template functions $(list-*).</div><div><br></div><div>We can reparse these elements back into the original message using a 3rd invocation of the json-parser.</div><div><br></div><div>Config:</div><div>@version: 4.0<br><br>log { <br> source { tcp(port(2000) flags(no-parse)); };<br><br> parser { json-parser(prefix(".json.")); };<br><br> parser { json-parser(prefix(".json.answers") template("${.json.answers}")); };<br><br> parser { json-parser(prefix(".json.answers_0_") template("$1")); };<br><br> destination { file("/tmp/json.out" template("$(format-flat-json --subkeys .json.)\n")); };<br>};</div><div><br></div><div><br></div><div>Trace:</div><div>[2022-06-10T18:30:42.820943] Setting value; name='.json.answers_0_name', value='_<a href="http://dnsaddr.bootstrap.libp2p.io">dnsaddr.bootstrap.libp2p.io</a>', type='string', msg='0x7ffff0014190', rcptid='104'<br>[2022-06-10T18:30:42.820970] Setting value; name='.json.answers_0_rdatatype', value='TXT', type='string', msg='0x7ffff0014190', rcptid='104'<br>[2022-06-10T18:30:42.820994] Setting value; name='.json.answers_0_ttl', value='600', type='int64', msg='0x7ffff0014190', rcptid='104'<br>[2022-06-10T18:30:42.821021] Setting value; name='.json.answers_0_rdata', value='dnsaddr=/dnsaddr/<a href="http://ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb">ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb</a>', type='string', msg='0x7ffff0014190', rcptid='104'<br></div><div><br></div><div>Getting somewhere. You can do this for each of your elements. The only issue is that you can't loop over the array. Yet.</div><div><br></div><div>So if you know there's a limited number of these elements, you can do this, by checking if $N is set, and do this parsing if it is.</div><div><br></div><div>I am afraid that's what we have at the moment. It is probably faster than doing it all in Python, but hey syslog-ng is not a programming language, is it? :)</div><div><br></div><div>I am giving a thought how we could do some level of iteration, but I don't have a very good idea at the moment.</div><div><br></div><div>Hope this helps,</div><div>Balazs<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 10, 2022 at 5:07 PM Александр Масленников <<a href="mailto:alexander.a.maslennikov@gmail.com">alexander.a.maslennikov@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">i'm not sure that *format-flat-json* works with lists same as with nested dicts.<div>My example uses format-flat-json, but without an additional filter on pothon, I was unable to flatten it<br></div><div>There is original message<br><br><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">{
"operation": "CLIENT_RESPONSE",
"identity": "<a href="http://ns-server.example.com" target="_blank">ns-server.example.com</a>",
"family": "INET",
"protocol": "TCP",
"query-ip": "192.168.xx.zz",
"query-port": "51000",
"response-ip": "192.168.yy.zz",
"response-port": "53",
"length": 691,
"rcode": "NOERROR",
"qname": "_<a href="http://dnsaddr.bootstrap.libp2p.io" target="_blank">dnsaddr.bootstrap.libp2p.io</a>",
"qtype": "TXT",
"latency": "0.000000",
"timestamp-rfc3339": "2022-06-06T08:47:58.797332215Z",
"answers": [{
"name": "_<a href="http://dnsaddr.bootstrap.libp2p.io" target="_blank">dnsaddr.bootstrap.libp2p.io</a>",
"rdatatype": "TXT",
"ttl": 600,
"rdata": "dnsaddr=/dnsaddr/<a href="http://ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb" target="_blank">ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb</a>"
}, {
"name": "_<a href="http://dnsaddr.bootstrap.libp2p.io" target="_blank">dnsaddr.bootstrap.libp2p.io</a>",
"rdatatype": "TXT",
"ttl": 600,
"rdata": "dnsaddr=/dnsaddr/<a href="http://sjc-1.bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN" target="_blank">sjc-1.bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN</a>"
}, {
"name": "_<a href="http://dnsaddr.bootstrap.libp2p.io" target="_blank">dnsaddr.bootstrap.libp2p.io</a>",
"rdatatype": "TXT",
"ttl": 600,
"rdata": "dnsaddr=/dnsaddr/<a href="http://ewr-1.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa" target="_blank">ewr-1.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa</a>"
}, {
"name": "_<a href="http://dnsaddr.bootstrap.libp2p.io" target="_blank">dnsaddr.bootstrap.libp2p.io</a>",
"rdatatype": "TXT",
"ttl": 600,
"rdata": "dnsaddr=/dnsaddr/<a href="http://ams-rust.bootstrap.libp2p.io/p2p/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS" target="_blank">ams-rust.bootstrap.libp2p.io/p2p/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS</a>"
}, {
"name": "_<a href="http://dnsaddr.bootstrap.libp2p.io" target="_blank">dnsaddr.bootstrap.libp2p.io</a>",
"rdatatype": "TXT",
"ttl": 600,
"rdata": "dnsaddr=/dnsaddr/<a href="http://sjc-2.bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp" target="_blank">sjc-2.bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp</a>"
}, {
"name": "_<a href="http://dnsaddr.bootstrap.libp2p.io" target="_blank">dnsaddr.bootstrap.libp2p.io</a>",
"rdatatype": "TXT",
"ttl": 600,
"rdata": "dnsaddr=/dnsaddr/<a href="http://nrt-1.bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt" target="_blank">nrt-1.bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt</a>"
}],
"country-isocode": "-"
}</pre>I would be very grateful to you if you have a solution using the built-in functions of syslog-ng.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">пт, 10 июн. 2022 г. в 15:00, <<a href="mailto:syslog-ng-request@lists.balabit.hu" target="_blank">syslog-ng-request@lists.balabit.hu</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send syslog-ng mailing list submissions to<br>
<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:syslog-ng-request@lists.balabit.hu" target="_blank">syslog-ng-request@lists.balabit.hu</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:syslog-ng-owner@lists.balabit.hu" target="_blank">syslog-ng-owner@lists.balabit.hu</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of syslog-ng digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. need help with parser to make flat nested json list of<br>
dictionaries (????????? ???????????)<br>
2. Re: need help with parser to make flat nested json list of<br>
dictionaries (Peter Kokai (pkokai))<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 10 Jun 2022 11:02:55 +0300<br>
From: ????????? ??????????? <<a href="mailto:alexander.a.maslennikov@gmail.com" target="_blank">alexander.a.maslennikov@gmail.com</a>><br>
To: <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br>
Subject: [syslog-ng] need help with parser to make flat nested json<br>
list of dictionaries<br>
Message-ID:<br>
<CA+G0nAjp1b6_50LbCROVPje1_B4R_AzNYiZ-_dT0m=<a href="mailto:fXcqwmHA@mail.gmail.com" target="_blank">fXcqwmHA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
hi all<br>
i have a json message that contains a nested json list of dicts<br>
<br>
{"a":1,"b":[{"c":1},{"c":2},{"c":3}]}<br>
<br>
i want to flat that message, so expected result looks like {<br>
"a": 1,<br>
"b_0_c": 1,<br>
"b_1_c": 2,<br>
"b_2_c": 3<br>
}<br>
<br>
My approach is a python implemented parser.<br>
Is it possible to achieve the same result using the built-in syslog-ng<br>
tools?<br>
My solution below<br>
<br>
@define kafka-implementation kafka-c<br>
<br>
python {<br>
<br>
import collections<br>
import json<br>
<br>
class FlattenedJson(object):<br>
<br>
def parse(self, log_message, flat_message=None):<br>
def flatten(d, parent_key='', sep='_'):<br>
items = []<br>
for k, v in d.items():<br>
new_key = parent_key + sep + k if parent_key else k<br>
if isinstance(v, collections.MutableMapping):<br>
items.extend(flatten(v, new_key, sep=sep).items())<br>
elif isinstance(v, list):<br>
for idx, value in enumerate(v):<br>
items.extend(flatten(value, new_key + sep +<br>
str(idx), sep).items())<br>
else:<br>
items.append((new_key, v))<br>
return dict(items)<br>
try:<br>
decoded_msg = json.loads(log_message['MESSAGE'].decode('utf-8'))<br>
flat_message = flatten(decoded_msg)<br>
final_message =<br>
str(json.dumps(flat_message)).encode(encoding='utf-8')<br>
log_message['MESSAGE'] = final_message<br>
except Exception as error:<br>
log_message['python_error'] = 'An exception occurred:<br>
{}'.format(error)<br>
return True<br>
};<br>
<br>
destination d_kafka_dnstap {<br>
kafka(<br>
topic("mytopic")<br>
bootstrap-servers("localhost:9092")<br>
message("$(format-flat-json --scope all-nv-pairs<br>
application_name=myapp @timestamp=${ISODATE} )")<br>
);<br>
};<br>
<br>
source s_net_dnstap { network( transport(udp) port(514) flags(no-parse) ); };<br>
<br>
parser p_dnstap { channel {<br>
parser { python(class("FlattenedJson")); };<br>
parser { json-parser(prefix("dnstap.")); };<br>
};<br>
};<br>
<br>
log { source(s_net_dnstap); parser(p_dnstap); destination(d_kafka_dnstap); };<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220610/0fb59c2a/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220610/0fb59c2a/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 10 Jun 2022 08:09:24 +0000<br>
From: "Peter Kokai (pkokai)" <<a href="mailto:Peter.Kokai@oneidentity.com" target="_blank">Peter.Kokai@oneidentity.com</a>><br>
To: "<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>" <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
Subject: Re: [syslog-ng] need help with parser to make flat nested<br>
json list of dictionaries<br>
Message-ID:<br>
<<a href="mailto:SA1PR19MB5641CF9A4E2AB1502C5AE348F8A69@SA1PR19MB5641.namprd19.prod.outlook.com" target="_blank">SA1PR19MB5641CF9A4E2AB1502C5AE348F8A69@SA1PR19MB5641.namprd19.prod.outlook.com</a>><br>
<br>
Content-Type: text/plain; charset="koi8-r"<br>
<br>
Hello,<br>
<br>
If the underlines are not a must in the key, yes you can use *format-flat-json* (it uses dot instead of underscore). It uses the same syntax as format-json.<br>
<br>
--<br>
Kokan<br>
<br>
________________________________________<br>
From: syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>> on behalf of ????????? ??????????? <<a href="mailto:alexander.a.maslennikov@gmail.com" target="_blank">alexander.a.maslennikov@gmail.com</a>><br>
Sent: 10 June 2022 10:02<br>
To: <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br>
Subject: [syslog-ng] need help with parser to make flat nested json list of dictionaries<br>
<br>
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
<br>
hi all<br>
i have a json message that contains a nested json list of dicts<br>
<br>
{"a":1,"b":[{"c":1},{"c":2},{"c":3}]}<br>
<br>
i want to flat that message, so expected result looks like {<br>
"a": 1,<br>
"b_0_c": 1,<br>
"b_1_c": 2,<br>
"b_2_c": 3<br>
}<br>
<br>
My approach is a python implemented parser.<br>
Is it possible to achieve the same result using the built-in syslog-ng tools?<br>
My solution below<br>
<br>
<br>
@define kafka-implementation kafka-c<br>
<br>
python {<br>
<br>
import collections<br>
import json<br>
<br>
class FlattenedJson(object):<br>
<br>
def parse(self, log_message, flat_message=None):<br>
def flatten(d, parent_key='', sep='_'):<br>
items = []<br>
for k, v in d.items():<br>
new_key = parent_key + sep + k if parent_key else k<br>
if isinstance(v, collections.MutableMapping):<br>
items.extend(flatten(v, new_key, sep=sep).items())<br>
elif isinstance(v, list):<br>
for idx, value in enumerate(v):<br>
items.extend(flatten(value, new_key + sep + str(idx), sep).items())<br>
else:<br>
items.append((new_key, v))<br>
return dict(items)<br>
try:<br>
decoded_msg = json.loads(log_message['MESSAGE'].decode('utf-8'))<br>
flat_message = flatten(decoded_msg)<br>
final_message = str(json.dumps(flat_message)).encode(encoding='utf-8')<br>
log_message['MESSAGE'] = final_message<br>
except Exception as error:<br>
log_message['python_error'] = 'An exception occurred: {}'.format(error)<br>
return True<br>
};<br>
<br>
destination d_kafka_dnstap {<br>
kafka(<br>
topic("mytopic")<br>
bootstrap-servers("localhost:9092")<br>
message("$(format-flat-json --scope all-nv-pairs application_name=myapp @timestamp=${ISODATE} )")<br>
);<br>
};<br>
<br>
source s_net_dnstap { network( transport(udp) port(514) flags(no-parse) ); };<br>
<br>
parser p_dnstap { channel {<br>
parser { python(class("FlattenedJson")); };<br>
parser { json-parser(prefix("dnstap.")); };<br>
};<br>
};<br>
<br>
log { source(s_net_dnstap); parser(p_dnstap); destination(d_kafka_dnstap); };<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
syslog-ng maillist - <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br>
<a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
<br>
<br>
------------------------------<br>
<br>
End of syslog-ng Digest, Vol 206, Issue 2<br>
*****************************************<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Bazsi</div></div>