Hi, You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elastic... Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.: https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsearch/blob/m... Regards, Balázs <syslog-ng-request@lists.balabit.hu> ezt írta (időpont: 2020. dec. 19., Szo, 13:00):
Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics:
1. elasticsearch and ILM (Russell Fulton)
---------- Forwarded message ---------- From: Russell Fulton <r.fulton@auckland.ac.nz> To: "Syslog-ng users' and developers' mailing list" < syslog-ng@lists.balabit.hu> Cc: Bcc: Date: Sat, 19 Dec 2020 01:40:42 +0000 Subject: [syslog-ng] elasticsearch and ILM I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on their forum is a bit erratic.
The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams.
So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”.
If anyone has a working set up that they are happy to share it would be wonderful.
Russell_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng