a question about the elasticsearch-http driver
hi all, I have a question about the elasticsearch-http driver. Is it possible to send events to a specific Elasticsearch ingest pipeline with this driver? Also, if sending events to ingest pipeline is possible, can the ingest pipeline name contain macros, so that the pipeline name can be set based on the data extracted from incoming events? Such functionality is implemented in rsyslog, and I was wondering if the same thing can be accomplished with syslog-ng. kind regards, risto
Hi Risto, Although I didn't test this myself, I see two ways you could achieve this. 1. By using the "pipeline" query parameter in the url e.g. url("https://localhost:9200/_bulk?pipeline=mypipeline") 2. By setting a default pipeline in the ES index I'm not sure however if elasticsearch-http()'s url parameter is templateable. Yoy'll have to check for yourself.
hi Fabien, and thanks for the tip! I have just tried the following destination driver where the pipeline name is set by the ${HOST} macro: destination d_elasticsearch_http { elasticsearch-http( index("syslog-ng") type("") url("http://localhost:9200/_bulk?pipeline=${HOST}") template("$(format-json --scope rfc5424 --scope dot-nv-pairs --rekey .* --shift 1 --scope nv-pairs --exclude DATE --key ISODATE @timestamp=${ISODATE})") ); }; It appears that the macros inside the url are not replaced with their values. Here is an example request that gets generated: POST /_bulk?pipeline=${HOST} HTTP/1.1 Host: localhost:9200 User-Agent: syslog-ng 3.32.1/libcurl 7.61.1 Accept: */* Content-Type: application/x-ndjson Content-Length: 296 ... So it seems that only constant pipeline names can be used, and one can not extract the pipeline name from the incoming event. kind regards, risto Kontakt Fabien Wernli (<wernli@in2p3.fr>) kirjutas kuupäeval R, 11. märts 2022 kell 23:11:
Hi Risto,
Although I didn't test this myself, I see two ways you could achieve this.
1. By using the "pipeline" query parameter in the url e.g. url("https://localhost:9200/_bulk?pipeline=mypipeline") 2. By setting a default pipeline in the ES index
I'm not sure however if elasticsearch-http()'s url parameter is templateable. Yoy'll have to check for yourself.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Risto, Thanks for the test. In my opinion, it would be flexible to allow templates in the URL, but also very dangerous. I'm not sure it is a good idea. If you only have a few pipelines, I'd rather use multiple elasticsearch destinations : one per pipeline. cheers
participants (2)
-
Fabien Wernli
-
Risto Vaarandi