<div dir="ltr">I'm still not getting it. My standard syslog data works fine is there something behind the scenes ? <div>My Syslog ES target wis working but getting nothing in the beats target no errors either. </div><div><br></div><div><div>destination d_es {</div><div> elasticsearch2(</div><div> disk-buffer(</div><div> reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option</div><div> dir("/opt/syslog-ng/buffer")</div><div> disk-buf-size(10485760)</div><div> mem-buf-length(100000) # number of messages stored in overflow queue</div><div> )</div><div> client-mode("http")</div><div> index("syslog-ng_${YEAR}.${MONTH}.${DAY}")</div><div> type("syslog") # Description: The type of the index. For example, type("test")</div><div> template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")</div><div> cluster-url("<a href="http://192.168.1.16:9200/">http://192.168.1.16:9200/</a>")</div><div> concurrent-requests("5") # Number of concurrrent batches</div><div> flush_limit("5000") # The number of messages in a single batch</div><div> skip-cluster-health-check("yes")</div><div> cluster("hal")</div><div> client_lib_dir("/usr/share/elasticsearch/lib")</div><div> );</div><div>};</div><div><br></div><div>destination d_es_beats {</div><div> elasticsearch2(</div><div> disk-buffer(</div><div> reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option</div><div> dir("/opt/syslog-ng/buffer/beats")</div><div> disk-buf-size(10485760)</div><div> mem-buf-length(100000) # number of messages stored in overflow queue</div><div> ) # END DiskBuffer</div><div> client-mode("http")</div><div> index("winlogbeat-${YEAR}.${MONTH}.${DAY}")</div><div> type("winlogbeat") # Description: The type of the index. For example, type("test")</div><div> #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")</div><div> #template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)")</div><div> template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)")</div><div> #template("${MESSAGE}")</div><div> cluster-url("<a href="http://192.168.1.16:9200/">http://192.168.1.16:9200/</a>")</div><div> concurrent-requests("5") # Number of concurrrent batches</div><div> flush_limit("5000") # The number of messages in a single batch</div><div> skip-cluster-health-check("yes")</div><div> cluster("hal")</div><div> client_lib_dir("/usr/share/elasticsearch/lib")</div><div> );</div><div>};</div></div><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 7:05 AM, Fabien Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<span class=""><br>
On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:<br>
> I don't get it, I don't have that in my current ES target for syslog.<br>
<br>
</span>Kibana and most other frontends and maybe even libraries use the query's<br>
time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${<wbr>DAY} in your<br>
example. They use UTC timezone to do that. If you use localtime, like in<br>
your example (implicit) documents having 00:30 as timestamp for example will<br>
end up in the wrong index, and this will also depend on DST.<br>
<br>
When you search for them in kibana you might end up with wrong<br>
results.<br>
<br>
TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you<br>
use time-based indices.<br>
<br>
Maybe we should even add that to the default SCL.<br>
<div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br></div>