[syslog-ng] Syslog-ng input for beats ?

[Scot]

I take that back.   Getting data in the winlogbeat target but the json is
not getting parsed.   All coming in as a messages block.
{
  "_index": "winlogbeat-2017.05.12",
  "_type": "winlogbeat",
  "_id": "AVv9Xfil6uwlymto3Hmd",
  "_score": null,
  "_source": {
    "SOURCE": "s_BEATS",
    "MESSAGE":
"{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@
http://192.168.1.16:9200
\",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\"
http://192.168.1.16:9200
\",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}",
    "HOST_FROM": "hal",
    "HOST": "hal",
    "@timestamp": "2017-05-12T11:54:03-04:00",
    "@message":
"{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@
http://192.168.1.16:9200
\",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\"
http://192.168.1.16:9200
\",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}"
  },
  "fields": {
    "@timestamp": [
      1494604443000
    ]
  },
  "sort": [
    1494604443000
  ]
}


On Fri, May 12, 2017 at 11:47 AM, Scot <scotrn at gmail.com> wrote:

> I'm still not getting it.  My standard syslog data works fine is there
> something behind the scenes ?
> My Syslog ES target wis working but getting nothing in the beats target no
> errors either.
>
> destination d_es {
>         elasticsearch2(
>                 disk-buffer(
>                 reliable(no) #  If set to no, the normal disk-buffer will
> be used. This provides a faster, option
>                 dir("/opt/syslog-ng/buffer")
>                 disk-buf-size(10485760)
>                 mem-buf-length(100000) # number of messages stored in
> overflow queue
>                 )
>                 client-mode("http")
>                 index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
>                 type("syslog") # Description: The type of the index. For
> example, type("test")
>                 template("$(format-json --scope rfc3164 --scope nv-pairs
> --exclude R_DATE --key ISODATE)\n")
>                 cluster-url("http://192.168.1.16:9200/")
>                 concurrent-requests("5") # Number of concurrrent batches
>                 flush_limit("5000") # The number of messages in a single
> batch
>                 skip-cluster-health-check("yes")
>                 cluster("hal")
>                 client_lib_dir("/usr/share/elasticsearch/lib")
>         );
> };
>
> destination d_es_beats {
>         elasticsearch2(
>                 disk-buffer(
>                 reliable(no) #  If set to no, the normal disk-buffer will
> be used. This provides a faster, option
>                 dir("/opt/syslog-ng/buffer/beats")
>                 disk-buf-size(10485760)
>                 mem-buf-length(100000) # number of messages stored in
> overflow queue
>                 ) # END DiskBuffer
>                 client-mode("http")
>                 index("winlogbeat-${YEAR}.${MONTH}.${DAY}")
>                 type("winlogbeat") # Description: The type of the index.
> For example, type("test")
>                 #template("$(format-json --scope rfc3164 --scope nv-pairs
> --exclude R_DATE --key ISODATE)\n")
>                 #template("$(format-json -s all-nv-pairs -p
> @timestamp=$ISODATE -p @message=$MESSAGE)")
>                 template("$(format-json -s all-nv-pairs -p
> @timestamp=$ISODATE -p @message=$MESSAGE)")
>                 #template("${MESSAGE}")
>                 cluster-url("http://192.168.1.16:9200/")
>                 concurrent-requests("5") # Number of concurrrent batches
>                 flush_limit("5000") # The number of messages in a single
> batch
>                 skip-cluster-health-check("yes")
>                 cluster("hal")
>                 client_lib_dir("/usr/share/elasticsearch/lib")
>         );
> };
>
>
> On Fri, May 12, 2017 at 7:05 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>
>> Hi,
>>
>> On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:
>> > I don't get it, I don't have that in my current ES target for syslog.
>>
>> Kibana and most other frontends and maybe even libraries use the query's
>> time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${DAY} in your
>> example. They use UTC timezone to do that. If you use localtime, like in
>> your example (implicit) documents having 00:30 as timestamp for example
>> will
>> end up in the wrong index, and this will also depend on DST.
>>
>> When you search for them in kibana you might end up with wrong
>> results.
>>
>> TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you
>> use time-based indices.
>>
>> Maybe we should even add that to the default SCL.
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170512/17afb231/attachment-0001.html>