[syslog-ng] parse rancher logs from syslog-ng files

[Laszlo Szemere (lszemere)]

Hello John,

> Parse json kv into diffeent fields to elastic and flatten the json fields.
1. If I understand your use case correctly, you can use the "marker" option of the json-parser, which is useful for only parse a specific message parts.
2. To flatten the json output, please refer to the "format-flat-json" template function: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.24/administration-guide/64#TOPIC-1298114 . It is available from 3.24, with: https://github.com/syslog-ng/syslog-ng/pull/2890


>  Use one field to create the index on destination elastic like "program" or custom added field on source side like "k8s-cluster-name"
The syntax for using macros in index name is:
  index("${MACRO_NAME}")
  index("foo-${MACRO_NAME}-bar")


I hope this will help.

Best regards,
Laci

________________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Tech Gurus <linuxgurus1 at gmail.com>
Sent: Monday, November 11, 2019 18:04
To: syslog-ng at lists.balabit.hu
Subject: [syslog-ng] parse rancher logs from syslog-ng files

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi ,

I have setup where rancher 2.3 send logs ogs to syslog-ng server , I am getting several GB per hour and write them to text file.

The log format are dynamic , general form are  [timestamp SourceContainer-host-IP Source-Program  list-of-nested-json-key/value-pairs] ..

I got the elasticsearch forwarding wokring with config below, however all json kv gets written to message field..


I need to be able to :

* Parse json kv into diffeent fields to elastic and flatten the json fields.
* Use one field to create the index on destination elastic like "program" or custom added field on source side like "k8s-cluster-name"


Thanks for help

John





destination d_elasticsearch_http {
    elasticsearch-http(
        url("https://elastricsearch:9200/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
        index("syslog-ng-test")
        #time-zone("UTC")
        type("")
        workers(4)
        batch-lines(100)
        #batch-timeout(10000)
        #timeout(10)
        tls(
            #ca-dir("/etc/syslog-ng/conf.d")
            ca-file("/etc/syslog-ng/conf.d/root-ca.pem")
            cert-file("/etc/syslog-ng/conf.d/admin.pem")
            key-file("/etc/syslog-ng/conf.d/admin.key")
            peer-verify(no)
        )
    );
};