Hi ,

I have setup where rancher 2.3 send logs ogs to syslog-ng server , I am getting several GB per hour and write them to text file.

The log format are dynamic , general form are [timestamp SourceContainer-host-IP Source-Program list-of-nested-json-key/value-pairs] ..

I got the elasticsearch forwarding wokring with config below, however all json kv gets written to message field..

I need to be able to :

* Parse json kv into diffeent fields to elastic and flatten the json fields.
* Use one field to create the index on destination elastic like "program" or custom added field on source side like "k8s-cluster-name"

Thanks for help

John

destination d_elasticsearch_http {
elasticsearch-http(
url("https://elastricsearch:9200/_bulk")
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE --key ISODATE @timestamp=${ISODATE})")
index("syslog-ng-test")
#time-zone("UTC")
type("")
workers(4)
batch-lines(100)
#batch-timeout(10000)
#timeout(10)
tls(
#ca-dir("/etc/syslog-ng/conf.d")
ca-file("/etc/syslog-ng/conf.d/root-ca.pem")
cert-file("/etc/syslog-ng/conf.d/admin.pem")
key-file("/etc/syslog-ng/conf.d/admin.key")
peer-verify(no)
)
);
};