[syslog-ng] a question about the elasticsearch-http driver

[Risto Vaarandi]

hi Fabien,

and thanks for the tip! I have just tried the following destination driver
where the pipeline name is set by the ${HOST} macro:

destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        url("http://localhost:9200/_bulk?pipeline=${HOST}")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};

It appears that the macros inside the url are not replaced with their
values. Here is an example request that gets generated:

POST /_bulk?pipeline=${HOST} HTTP/1.1
Host: localhost:9200
User-Agent: syslog-ng 3.32.1/libcurl 7.61.1
Accept: */*
Content-Type: application/x-ndjson
Content-Length: 296
...

So it seems that only constant pipeline names can be used, and one can not
extract the pipeline name from the incoming event.

kind regards,
risto

Kontakt Fabien Wernli (<wernli at in2p3.fr>) kirjutas kuupäeval R, 11. märts
2022 kell 23:11:

> Hi Risto,
>
> Although I didn't test this myself, I see two ways you could achieve this.
>
> 1. By using the "pipeline" query parameter in the url
>    e.g. url("https://localhost:9200/_bulk?pipeline=mypipeline")
> 2. By setting a default pipeline in the ES index
>
> I'm not sure however if elasticsearch-http()'s url parameter is
> templateable. Yoy'll have to check for yourself.
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220312/ff1c888d/attachment.htm>