I take that back. Getting data in the winlogbeat target but the json is not getting parsed. All coming in as a messages block. { "_index": "winlogbeat-2017.05.12", "_type": "winlogbeat", "_id": "AVv9Xfil6uwlymto3Hmd", "_score": null, "_source": { "SOURCE": "s_BEATS", "MESSAGE": "{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@ http://192.168.1.16:9200 \",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\" http://192.168.1.16:9200 \",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}", "HOST_FROM": "hal", "HOST": "hal", "@timestamp": "2017-05-12T11:54:03-04:00", "@message": "{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@ http://192.168.1.16:9200 \",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\" http://192.168.1.16:9200 \",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}" }, "fields": { "@timestamp": [ 1494604443000 ] }, "sort": [ 1494604443000 ] } On Fri, May 12, 2017 at 11:47 AM, Scot <scotrn@gmail.com> wrote:
I'm still not getting it. My standard syslog data works fine is there something behind the scenes ? My Syslog ES target wis working but getting nothing in the beats target no errors either.
destination d_es { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); };
destination d_es_beats { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer/beats") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) # END DiskBuffer client-mode("http") index("winlogbeat-${YEAR}.${MONTH}.${DAY}") type("winlogbeat") # Description: The type of the index. For example, type("test") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") #template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)") template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)") #template("${MESSAGE}") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); };
On Fri, May 12, 2017 at 7:05 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:
I don't get it, I don't have that in my current ES target for syslog.
Kibana and most other frontends and maybe even libraries use the query's time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${DAY} in your example. They use UTC timezone to do that. If you use localtime, like in your example (implicit) documents having 00:30 as timestamp for example will end up in the wrong index, and this will also depend on DST.
When you search for them in kibana you might end up with wrong results.
TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you use time-based indices.
Maybe we should even add that to the default SCL.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq