<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Russell wishes to delete this post :)
<div class=""><br class="">
</div>
<div class="">Russell did what he should have done before posting! He ran tcpdump and found the error response from ES!</div>
<div class=""><br class="">
</div>
<div class="">Apologies for the noise. The original question is still open!<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 20/12/2020, at 10:38 AM, Russell Fulton <<a href="mailto:r.fulton@auckland.ac.nz" class="">r.fulton@auckland.ac.nz</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="color: #ff0000;" class="">Caution - Forged Internal Domain!<br class="">
This e-mail cannot be validated and may not have been sent by the sender shown in the 'From' field.<br class="">
If you were not expecting to receive this e-mail we recommend you contact the sender to confirm that they sent it.<br class="">
If you believe this email was legitimately sent, we suggest the sender notify the Staff Service Centre that it has been received as a forged (fake) e-mail.<br class="">
Please contact the Staff Service Centre on extension 86000 if you require further assistance.<br class="">
</span><br class="">
<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 19/12/2020, at 2:40 PM, Russell Fulton <<a href="mailto:r.fulton@auckland.ac.nz" class="">r.fulton@auckland.ac.nz</a>> wrote:</div>
<div class="">
<div class=""><br class="">
I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on
their forum is a bit erratic. <br class="">
<br class="">
The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams.<br class="">
<br class="">
So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”.
<br class="">
<br class="">
If anyone has a working set up that they are happy to share it would be wonderful.<br class="">
<br class="">
Russell<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div class="">This is getting weirder !</div>
<div class=""><br class="">
</div>
<div class="">ES is now failing to create my new index at all in spite of the destination definition being practically identical to one that works:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(45, 150, 30);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">destination d_auth_elastic</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> {</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(193, 101, 28);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures" class="">elasticsearch_http(</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> index(</span><span style="font-variant-ligatures: no-common-ligatures" class="">"auth_${YEAR}.${MONTH}.${DAY}"</span><span style="font-variant-ligatures: no-common-ligatures;" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> type(</span><span style="font-variant-ligatures: no-common-ligatures; color: #9d206f" class="">"_doc"</span><span style="font-variant-ligatures: no-common-ligatures" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> persist-name(</span><span style="font-variant-ligatures: no-common-ligatures; color: #9d206f" class="">"auth"</span><span style="font-variant-ligatures: no-common-ligatures" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> template(</span><span style="font-variant-ligatures: no-common-ligatures" class="">"$(format-json --scope nv-pairs --exclude HOST_FROM
</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude HOST </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude srcip* </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude SOURCE </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude PROGRAM </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude 0 </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude 1 </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude PID </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude LEGACY_MSGHDR </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --key ISODATE)\n"</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span>
<span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> url(</span><span style="font-variant-ligatures: no-common-ligatures" class="">"<a href="http://secesprd01.its.auckland.ac.nz:9200/_bulk" class="">http://secesprd01.its.auckland.ac.nz:9200/_bulk</a>"</span><span style="font-variant-ligatures: no-common-ligatures;" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(193, 101, 28);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">};</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(45, 150, 30);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">destination d_authm_elastic</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> {</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(193, 101, 28);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures" class="">elasticsearch_http(</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> index(</span><span style="font-variant-ligatures: no-common-ligatures" class="">"auth-000001"</span><span style="font-variant-ligatures: no-common-ligatures;" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> type(</span><span style="font-variant-ligatures: no-common-ligatures; color: #9d206f" class="">""</span><span style="font-variant-ligatures: no-common-ligatures" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> template(</span><span style="font-variant-ligatures: no-common-ligatures" class="">"$(format-json --scope nv-pairs --exclude HOST_FROM
</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude HOST </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude srcip* </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude SOURCE </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude PROGRAM </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude 0 </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude 1 </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude PID </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> --exclude LEGACY_MSGHDR )\n"</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span>
<span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(157, 32, 111);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> url(</span><span style="font-variant-ligatures: no-common-ligatures" class="">"<a href="http://secesprd01.its.auckland.ac.nz:9200/_bulk" class="">http://secesprd01.its.auckland.ac.nz:9200/_bulk</a>"</span><span style="font-variant-ligatures: no-common-ligatures;" class="">)</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(193, 101, 28);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">};</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(45, 150, 30);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">log</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> {</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">source(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> s_loghost );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">parser(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> p_patterns );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">filter(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> f_classified );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">filter(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> f_ping999 );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">filter(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> f_forti_stats
);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">rewrite(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> r_user1 );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">rewrite(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> r_user2 );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">rewrite(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> r_srcip );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">parser(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> p_srcip );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">rewrite(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> r_srcip_country
);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">rewrite(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> r_srcip_city);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">rewrite(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> r_src_loc);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">rewrite(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> r_ISODATE );</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">destination(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> d_auth_elastic
);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">destination(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> d_authm_elastic);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #c1651c" class="">flags(</span><span style="font-variant-ligatures: no-common-ligatures" class=""> flow-control);</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(193, 101, 28);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">};</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; min-height: 13px;" class="">
<span style="font-family: Menlo;" class="">d_auth_elastic </span>works fine but <span style="font-family: Menlo;" class="">d_authm_elastic
</span>never gets created. No errors on the ES end.</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; min-height: 13px;" class="">
Anyone have any clues as to what is going on?</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; min-height: 13px;" class="">
I am at the point of starting tcpdump and looking at what is going over the network.</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; min-height: 13px;" class="">
</div>
</div>
<br class="">
</div>
______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>