[syslog-ng] Parsing Messages for Elasticsearch

21 Dec 2016

      Hi,

I get a lot of name="value" formated messages which I want to receive in syslog-ng and pass them into elasticsearch via the elasticsearch2 module.

The message passing works fine, however I'm not able to parse the messages for elasticsearch yet.
What I do get in elasticsearch is the whole syslog as a string in the message, what I actually want is all the keys as fields with the value.

My elasticsearch destination is configured as followed:

destination d_elastic {
    elasticsearch2(
        client-lib-dir("/usr/share/elastic-5-lib/lib")
        client-mode("http")
        cluster("ng")
        index("ng-${YEAR}.${MONTH}.${DAY}")
        type("syslog")
        cluster-url("http://172.18.1.5:9200/")
        template("$(format-json --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
        flush-limit("100")
        concurrent-requests("10")
        disk-buffer(
            disk-buf-size(500000000)
            dir("/opt/disk-buffer")
            reliable(yes)
        )
    );
};

I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all?

Thanks
Tim

[syslog-ng] Parsing Messages for Elasticsearch

Jentz, Tim