<div dir="ltr">I take that back. Getting data in the winlogbeat target but the json is not getting parsed. All coming in as a messages block. <div><div>{</div><div> "_index": "winlogbeat-2017.05.12",</div><div> "_type": "winlogbeat",</div><div> "_id": "AVv9Xfil6uwlymto3Hmd",</div><div> "_score": null,</div><div> "_source": {</div><div> "SOURCE": "s_BEATS",</div><div> "MESSAGE": "{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>\",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\"<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>\",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}",</div><div> "HOST_FROM": "hal",</div><div> "HOST": "hal",</div><div> "@timestamp": "2017-05-12T11:54:03-04:00",</div><div> "@message": "{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>\",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\"<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>\",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}"</div><div> },</div><div> "fields": {</div><div> "@timestamp": [</div><div> 1494604443000</div><div> ]</div><div> },</div><div> "sort": [</div><div> 1494604443000</div><div> ]</div><div>}</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 11:47 AM, Scot <span dir="ltr"><<a href="mailto:scotrn@gmail.com" target="_blank">scotrn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm still not getting it. My standard syslog data works fine is there something behind the scenes ? <div>My Syslog ES target wis working but getting nothing in the beats target no errors either. </div><div><br></div><div><span class=""><div>destination d_es {</div><div> elasticsearch2(</div><div> disk-buffer(</div><div> reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option</div><div> dir("/opt/syslog-ng/buffer")</div><div> disk-buf-size(10485760)</div><div> mem-buf-length(100000) # number of messages stored in overflow queue</div><div> )</div><div> client-mode("http")</div><div> index("syslog-ng_${YEAR}.${<wbr>MONTH}.${DAY}")</div><div> type("syslog") # Description: The type of the index. For example, type("test")</div><div> template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")</div><div> cluster-url("<a href="http://192.168.1.16:9200/" target="_blank">http://192.168.1.<wbr>16:9200/</a>")</div><div> concurrent-requests("5") # Number of concurrrent batches</div><div> flush_limit("5000") # The number of messages in a single batch</div><div> skip-cluster-health-check("<wbr>yes")</div><div> cluster("hal")</div><div> client_lib_dir("/usr/share/<wbr>elasticsearch/lib")</div><div> );</div><div>};</div><div><br></div></span><div>destination d_es_beats {</div><span class=""><div> elasticsearch2(</div><div> disk-buffer(</div><div> reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option</div></span><div> dir("/opt/syslog-ng/buffer/<wbr>beats")</div><span class=""><div> disk-buf-size(10485760)</div><div> mem-buf-length(100000) # number of messages stored in overflow queue</div></span><span class=""><div> ) # END DiskBuffer</div><div> client-mode("http")</div><div> index("winlogbeat-${YEAR}.${<wbr>MONTH}.${DAY}")</div><div> type("winlogbeat") # Description: The type of the index. For example, type("test")</div><div> #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")</div><div> #template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)")</div><div> template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)")</div></span><div> #template("${MESSAGE}")</div><span class=""><div> cluster-url("<a href="http://192.168.1.16:9200/" target="_blank">http://192.168.1.<wbr>16:9200/</a>")</div><div> concurrent-requests("5") # Number of concurrrent batches</div><div> flush_limit("5000") # The number of messages in a single batch</div><div> skip-cluster-health-check("<wbr>yes")</div><div> cluster("hal")</div><div> client_lib_dir("/usr/share/<wbr>elasticsearch/lib")</div><div> );</div><div>};</div></span></div><div><div><br></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 7:05 AM, Fabien Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<span><br>
On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:<br>
> I don't get it, I don't have that in my current ES target for syslog.<br>
<br>
</span>Kibana and most other frontends and maybe even libraries use the query's<br>
time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${D<wbr>AY} in your<br>
example. They use UTC timezone to do that. If you use localtime, like in<br>
your example (implicit) documents having 00:30 as timestamp for example will<br>
end up in the wrong index, and this will also depend on DST.<br>
<br>
When you search for them in kibana you might end up with wrong<br>
results.<br>
<br>
TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you<br>
use time-based indices.<br>
<br>
Maybe we should even add that to the default SCL.<br>
<div class="m_6723053756094038939HOEnZb"><div class="m_6723053756094038939h5"><br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>