[syslog-ng] parse rancher logs from syslog-ng files

[Tech Gurus]

Hi ,

I have setup where rancher 2.3 send logs ogs to syslog-ng server , I am
getting several GB per hour and write them to text file.

The log format are dynamic , general form are  [timestamp
SourceContainer-host-IP Source-Program
 list-of-nested-json-key/value-pairs] ..

I got the elasticsearch forwarding wokring with config below, however all
json kv gets written to message field..


I need to be able to :

* Parse json kv into diffeent fields to elastic and flatten the json
fields.
* Use one field to create the index on destination elastic like "program"
or custom added field on source side like "k8s-cluster-name"


Thanks for help

John





destination d_elasticsearch_http {
    elasticsearch-http(
        url("https://elastricsearch:9200/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
        index("syslog-ng-test")
        #time-zone("UTC")
        type("")
        workers(4)
        batch-lines(100)
        #batch-timeout(10000)
        #timeout(10)
        tls(
            #ca-dir("/etc/syslog-ng/conf.d")
            ca-file("/etc/syslog-ng/conf.d/root-ca.pem")
            cert-file("/etc/syslog-ng/conf.d/admin.pem")
            key-file("/etc/syslog-ng/conf.d/admin.key")
            peer-verify(no)
        )
    );
};
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20191111/47cd31de/attachment.html>