Re: [syslog-ng] elasticsearch and ILM

19 Dec 2020

      On 19/12/2020, at 2:40 PM, Russell Fulton <r.fulton@auckland.ac.nz<mailto:r.fulton@auckland.ac.nz>> wrote:

I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on their forum is a bit erratic.

The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams.

So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”.

If anyone has a working set up that they are happy to share it would be wonderful.

Russell

This is getting weirder !

ES is now failing to create my new index at all in spite of the destination definition being practically identical to one that works:

destination d_auth_elastic {
  elasticsearch_http(
    index("auth_${YEAR}.${MONTH}.${DAY}")
   type("_doc")
   persist-name("auth")
   template("$(format-json --scope nv-pairs --exclude HOST_FROM
                                            --exclude HOST
                                            --exclude srcip*
                                            --exclude SOURCE
                                            --exclude PROGRAM
                                            --exclude 0
                                            --exclude 1
                                            --exclude PID
                                            --exclude LEGACY_MSGHDR
                         --key ISODATE)\n"
         )

    url("http://secesprd01.its.auckland.ac.nz:9200/_bulk")
  );
};

destination d_authm_elastic {
  elasticsearch_http(
    index("auth-000001")
   type("")
   template("$(format-json --scope nv-pairs --exclude HOST_FROM
                                            --exclude HOST
                                            --exclude srcip*
                                            --exclude SOURCE
                                            --exclude PROGRAM
                                            --exclude 0
                                            --exclude 1
                                            --exclude PID
                                            --exclude LEGACY_MSGHDR )\n"
         )

    url("http://secesprd01.its.auckland.ac.nz:9200/_bulk")
  );
};

log {
    source( s_loghost );
    parser( p_patterns );
    filter( f_classified );
    filter( f_ping999 );
    filter( f_forti_stats );
    rewrite( r_user1 );
    rewrite( r_user2 );
    rewrite( r_srcip );
    parser( p_srcip );
    rewrite( r_srcip_country );
    rewrite( r_srcip_city);
    rewrite( r_src_loc);
    rewrite( r_ISODATE );
    destination( d_auth_elastic );
    destination( d_authm_elastic);
    flags( flow-control);
};

d_auth_elastic works fine but d_authm_elastic never gets created.  No errors on the ES end.

Anyone have any clues as to what is going on?

I am at the point of starting tcpdump and looking at what is going over the network.