[syslog-ng] elasticsearch and ILM
Németh Balázs
jobbara.artalmatlan at gmail.com
Sun Dec 20 12:43:15 UTC 2020
Hi,
Originally I responsed to a digest message so you might did not get it.
You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and
adjust the http() block's body part to use create, instead of index.
- body("$(format-json --scope none --omit-empty-values index._index=`index`
index._type=`type` index._id=`custom_id`)\n`template`")
+ body("$(format-json --scope none --omit-empty-values
create._index=`index` index._type=`type`
index._id=`custom_id`)\n`template`")
https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elastic-http.conf
Although this will make it work with data streams you will still have to
take care of everything else just as with now called legacy indexes
(templates, mapping).
Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in
flux.
ILM is mostly used for cases when you want to replace indexes according to
storage tiers in multi node clusters (warm, hot, cold node) or want to have
delete indexes older than a specified time.
If you just need a separate index for example every month, then use the
data macros in the elasticsearch-http destination when you specify the
index name.
/ Warning
I do not work for Elastic, these are my personal experiences.
ILM highly depends on things like rollover alias, which is set by the index
template - created by beats in advance - whenever a new index is created.
And it is versioned.
Elasticsearch - the vendor - puts a lot of effort into making beats +
Elasticsearch nodes themselves to work together.
However they always change something upon upgrades which requires great
effort to accomodate in syslog-ng.
If you use syslog-ng with Elasticsearch, I highly recommend to go full
manually with everything, maybe add enrichments with Logstash.
You should not mix things used by beats with syslog-ng (templates, mapping,
ILM, pipelines, dashboards, etc).
/
If you need some maintained examples, have a look at my
configs.:https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsearch/blob/master/elasticsearch/template-network.json
Regards,
Balázs
>
> ---------- Forwarded message ----------
> From: Russell Fulton <r.fulton at auckland.ac.nz>
> To: "Syslog-ng users' and developers' mailing list" <
> syslog-ng at lists.balabit.hu>
> Cc:
> Bcc:
> Date: Sat, 19 Dec 2020 21:47:12 +0000
> Subject: Re: [syslog-ng] elasticsearch and ILM
> Russell wishes to delete this post :)
>
> Russell did what he should have done before posting! He ran tcpdump and
> found the error response from ES!
>
> Apologies for the noise. The original question is still open!
>
> On 20/12/2020, at 10:38 AM, Russell Fulton <r.fulton at auckland.ac.nz>
> wrote:
>
> Caution - Forged Internal Domain!
> This e-mail cannot be validated and may not have been sent by the sender
> shown in the 'From' field.
> If you were not expecting to receive this e-mail we recommend you contact
> the sender to confirm that they sent it.
> If you believe this email was legitimately sent, we suggest the sender
> notify the Staff Service Centre that it has been received as a forged
> (fake) e-mail.
> Please contact the Staff Service Centre on extension 86000 if you require
> further assistance.
>
>
>
> On 19/12/2020, at 2:40 PM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:
>
> I have been using the elasticsearch_http driver to push stuff into ES for
> well over a year. Now I am trying to use the Index Lifecycle Management
> (ILM) to manage these indexes and finding it very confusing. There seem to
> be holes in the ES docs and help on their forum is a bit erratic.
>
> The most confusing thing is the “rollover index alias” and I have yet to
> find a coherent explanation of exactly what it does and how to set it. Most
> of the documentation seems to assume you are using filebeat, logstash or
> datastreams.
>
> So far as I can see you syslog-ng can not write to datastreams as you need
> to use “create” rather than “index”.
>
> If anyone has a working set up that they are happy to share it would be
> wonderful.
>
> Russell
>
>
> This is getting weirder !
>
> ES is now failing to create my new index at all in spite of the
> destination definition being practically identical to one that works:
>
> destination d_auth_elastic {
> elasticsearch_http(
> index("auth_${YEAR}.${MONTH}.${DAY}")
> type("_doc")
> persist-name("auth")
> template("$(format-json --scope nv-pairs --exclude HOST_FROM
>
>
> --exclude HOST
>
>
> --exclude srcip*
>
>
> --exclude SOURCE
>
>
> --exclude PROGRAM
>
>
> --exclude 0
>
>
> --exclude 1
>
>
> --exclude PID
>
>
> --exclude LEGACY_MSGHDR
>
>
> --key ISODATE)\n"
> )
>
> url("http://secesprd01.its.auckland.ac.nz:9200/_bulk")
> );
> };
>
> destination d_authm_elastic {
> elasticsearch_http(
> index("auth-000001")
> type("")
> template("$(format-json --scope nv-pairs --exclude HOST_FROM
>
>
> --exclude HOST
>
>
> --exclude srcip*
>
>
> --exclude SOURCE
>
>
> --exclude PROGRAM
>
>
> --exclude 0
>
>
> --exclude 1
>
>
> --exclude PID
>
>
> --exclude LEGACY_MSGHDR )\n"
> )
>
> url("http://secesprd01.its.auckland.ac.nz:9200/_bulk")
> );
> };
>
>
>
> log {
> source( s_loghost );
> parser( p_patterns );
> filter( f_classified );
> filter( f_ping999 );
> filter( f_forti_stats );
> rewrite( r_user1 );
> rewrite( r_user2 );
> rewrite( r_srcip );
> parser( p_srcip );
> rewrite( r_srcip_country );
> rewrite( r_srcip_city);
> rewrite( r_src_loc);
> rewrite( r_ISODATE );
> destination( d_auth_elastic );
> destination( d_authm_elastic);
> flags( flow-control);
> };
>
> d_auth_elastic works fine but d_authm_elastic never gets created. No
> errors on the ES end.
>
> Anyone have any clues as to what is going on?
>
> I am at the point of starting tcpdump and looking at what is going over
> the network.
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
> _______________________________________________
> syslog-ng maillist - syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201220/d1960a44/attachment-0001.html>
More information about the syslog-ng
mailing list