i'm not sure that *format-flat-json* works with lists same as with nested dicts.

My example uses format-flat-json, but without an additional filter on pothon, I was unable to flatten it

There is original message

{
	"operation": "CLIENT_RESPONSE",
	"identity": "ns-server.example.com",
	"family": "INET",
	"protocol": "TCP",
	"query-ip": "192.168.xx.zz",
	"query-port": "51000",
	"response-ip": "192.168.yy.zz",
	"response-port": "53",
	"length": 691,
	"rcode": "NOERROR",
	"qname": "_dnsaddr.bootstrap.libp2p.io",
	"qtype": "TXT",
	"latency": "0.000000",
	"timestamp-rfc3339": "2022-06-06T08:47:58.797332215Z",
	"answers": [{
		"name": "_dnsaddr.bootstrap.libp2p.io",
		"rdatatype": "TXT",
		"ttl": 600,
		"rdata": "dnsaddr=/dnsaddr/ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb"
	}, {
		"name": "_dnsaddr.bootstrap.libp2p.io",
		"rdatatype": "TXT",
		"ttl": 600,
		"rdata": "dnsaddr=/dnsaddr/sjc-1.bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN"
	}, {
		"name": "_dnsaddr.bootstrap.libp2p.io",
		"rdatatype": "TXT",
		"ttl": 600,
		"rdata": "dnsaddr=/dnsaddr/ewr-1.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
	}, {
		"name": "_dnsaddr.bootstrap.libp2p.io",
		"rdatatype": "TXT",
		"ttl": 600,
		"rdata": "dnsaddr=/dnsaddr/ams-rust.bootstrap.libp2p.io/p2p/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS"
	}, {
		"name": "_dnsaddr.bootstrap.libp2p.io",
		"rdatatype": "TXT",
		"ttl": 600,
		"rdata": "dnsaddr=/dnsaddr/sjc-2.bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp"
	}, {
		"name": "_dnsaddr.bootstrap.libp2p.io",
		"rdatatype": "TXT",
		"ttl": 600,
		"rdata": "dnsaddr=/dnsaddr/nrt-1.bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
	}],
	"country-isocode": "-"
}

I would be very grateful to you if you have a solution using the built-in functions of syslog-ng.

пт, 10 июн. 2022 г. в 15:00, <syslog-ng-request@lists.balabit.hu>:

Send syslog-ng mailing list submissions to
        syslog-ng@lists.balabit.hu

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
        syslog-ng-request@lists.balabit.hu

You can reach the person managing the list at
        syslog-ng-owner@lists.balabit.hu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."


Today's Topics:

   1.  need help with parser to make flat nested json list of
      dictionaries (????????? ???????????)
   2. Re:  need help with parser to make flat nested json list of
      dictionaries (Peter Kokai (pkokai))


----------------------------------------------------------------------

Message: 1
Date: Fri, 10 Jun 2022 11:02:55 +0300
From: ????????? ???????????  <alexander.a.maslennikov@gmail.com>
To: syslog-ng@lists.balabit.hu
Subject: [syslog-ng] need help with parser to make flat nested json
        list of dictionaries
Message-ID:
        <CA+G0nAjp1b6_50LbCROVPje1_B4R_AzNYiZ-_dT0m=fXcqwmHA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

hi all
i have a json message that contains a nested json list of dicts

{"a":1,"b":[{"c":1},{"c":2},{"c":3}]}

i want to flat that message, so expected result looks like {
"a": 1,
"b_0_c": 1,
"b_1_c": 2,
"b_2_c": 3
}

My approach is a python implemented parser.
Is it possible to achieve the same result using the built-in syslog-ng
tools?
My solution below

@define kafka-implementation kafka-c

python {

import collections
import json

class FlattenedJson(object):

    def parse(self, log_message, flat_message=None):
        def flatten(d, parent_key='', sep='_'):
            items = []
            for k, v in d.items():
                new_key = parent_key + sep + k if parent_key else k
                if isinstance(v, collections.MutableMapping):
                    items.extend(flatten(v, new_key, sep=sep).items())
                elif isinstance(v, list):
                    for idx, value in enumerate(v):
                        items.extend(flatten(value, new_key + sep +
str(idx), sep).items())
                else:
                    items.append((new_key, v))
            return dict(items)
        try:
            decoded_msg = json.loads(log_message['MESSAGE'].decode('utf-8'))
            flat_message = flatten(decoded_msg)
            final_message =
str(json.dumps(flat_message)).encode(encoding='utf-8')
            log_message['MESSAGE'] = final_message
        except Exception as error:
            log_message['python_error'] = 'An exception occurred:
{}'.format(error)
        return True
};

destination d_kafka_dnstap {
  kafka(
    topic("mytopic")
    bootstrap-servers("localhost:9092")
    message("$(format-flat-json  --scope all-nv-pairs
application_name=myapp @timestamp=${ISODATE} )")
  );
};

source s_net_dnstap { network( transport(udp) port(514) flags(no-parse) ); };

parser p_dnstap { channel {
    parser { python(class("FlattenedJson")); };
    parser { json-parser(prefix("dnstap.")); };
  };
};

log { source(s_net_dnstap); parser(p_dnstap); destination(d_kafka_dnstap); };
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220610/0fb59c2a/attachment-0001.htm>

------------------------------

Message: 2
Date: Fri, 10 Jun 2022 08:09:24 +0000
From: "Peter Kokai (pkokai)" <Peter.Kokai@oneidentity.com>
To: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu>
Subject: Re: [syslog-ng] need help with parser to make flat nested
        json list of dictionaries
Message-ID:
        <SA1PR19MB5641CF9A4E2AB1502C5AE348F8A69@SA1PR19MB5641.namprd19.prod.outlook.com>

Content-Type: text/plain; charset="koi8-r"

Hello,

If the underlines are not a must in the key, yes you can use *format-flat-json* (it uses dot instead of underscore). It uses the same syntax as format-json.

--
Kokan

________________________________________
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of ????????? ??????????? <alexander.a.maslennikov@gmail.com>
Sent: 10 June 2022 10:02
To: syslog-ng@lists.balabit.hu
Subject: [syslog-ng] need help with parser to make flat nested json list of dictionaries

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

hi all
i have a json message that contains a nested json list of dicts

{"a":1,"b":[{"c":1},{"c":2},{"c":3}]}

i want to flat that message, so expected result looks like {
"a": 1,
"b_0_c": 1,
"b_1_c": 2,
"b_2_c": 3
}

My approach is a python implemented parser.
Is it possible to achieve the same result using the built-in syslog-ng tools?
My solution below


@define kafka-implementation kafka-c

python {

import collections
import json

class FlattenedJson(object):

    def parse(self, log_message, flat_message=None):
        def flatten(d, parent_key='', sep='_'):
            items = []
            for k, v in d.items():
                new_key = parent_key + sep + k if parent_key else k
                if isinstance(v, collections.MutableMapping):
                    items.extend(flatten(v, new_key, sep=sep).items())
                elif isinstance(v, list):
                    for idx, value in enumerate(v):
                        items.extend(flatten(value, new_key + sep + str(idx), sep).items())
                else:
                    items.append((new_key, v))
            return dict(items)
        try:
            decoded_msg = json.loads(log_message['MESSAGE'].decode('utf-8'))
            flat_message = flatten(decoded_msg)
            final_message = str(json.dumps(flat_message)).encode(encoding='utf-8')
            log_message['MESSAGE'] = final_message
        except Exception as error:
            log_message['python_error'] = 'An exception occurred: {}'.format(error)
        return True
};

destination d_kafka_dnstap {
  kafka(
    topic("mytopic")
    bootstrap-servers("localhost:9092")
    message("$(format-flat-json  --scope all-nv-pairs application_name=myapp @timestamp=${ISODATE} )")
  );
};

source s_net_dnstap { network( transport(udp) port(514) flags(no-parse) ); };

parser p_dnstap { channel {
    parser { python(class("FlattenedJson")); };
    parser { json-parser(prefix("dnstap.")); };
  };
};

log { source(s_net_dnstap); parser(p_dnstap); destination(d_kafka_dnstap); };


------------------------------

Subject: Digest Footer

_______________________________________________
syslog-ng maillist  -  syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng


------------------------------

End of syslog-ng Digest, Vol 206, Issue 2
*****************************************

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq