[syslog-ng] Syslog-ng input for beats ?

[Scot]

I'm still not getting it.  My standard syslog data works fine is there
something behind the scenes ?
My Syslog ES target wis working but getting nothing in the beats target no
errors either.

destination d_es {
        elasticsearch2(
                disk-buffer(
                reliable(no) #  If set to no, the normal disk-buffer will
be used. This provides a faster, option
                dir("/opt/syslog-ng/buffer")
                disk-buf-size(10485760)
                mem-buf-length(100000) # number of messages stored in
overflow queue
                )
                client-mode("http")
                index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
                type("syslog") # Description: The type of the index. For
example, type("test")
                template("$(format-json --scope rfc3164 --scope nv-pairs
--exclude R_DATE --key ISODATE)\n")
                cluster-url("http://192.168.1.16:9200/")
                concurrent-requests("5") # Number of concurrrent batches
                flush_limit("5000") # The number of messages in a single
batch
                skip-cluster-health-check("yes")
                cluster("hal")
                client_lib_dir("/usr/share/elasticsearch/lib")
        );
};

destination d_es_beats {
        elasticsearch2(
                disk-buffer(
                reliable(no) #  If set to no, the normal disk-buffer will
be used. This provides a faster, option
                dir("/opt/syslog-ng/buffer/beats")
                disk-buf-size(10485760)
                mem-buf-length(100000) # number of messages stored in
overflow queue
                ) # END DiskBuffer
                client-mode("http")
                index("winlogbeat-${YEAR}.${MONTH}.${DAY}")
                type("winlogbeat") # Description: The type of the index.
For example, type("test")
                #template("$(format-json --scope rfc3164 --scope nv-pairs
--exclude R_DATE --key ISODATE)\n")
                #template("$(format-json -s all-nv-pairs -p
@timestamp=$ISODATE -p @message=$MESSAGE)")
                template("$(format-json -s all-nv-pairs -p
@timestamp=$ISODATE -p @message=$MESSAGE)")
                #template("${MESSAGE}")
                cluster-url("http://192.168.1.16:9200/")
                concurrent-requests("5") # Number of concurrrent batches
                flush_limit("5000") # The number of messages in a single
batch
                skip-cluster-health-check("yes")
                cluster("hal")
                client_lib_dir("/usr/share/elasticsearch/lib")
        );
};


On Fri, May 12, 2017 at 7:05 AM, Fabien Wernli <wernli at in2p3.fr> wrote:

> Hi,
>
> On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:
> > I don't get it, I don't have that in my current ES target for syslog.
>
> Kibana and most other frontends and maybe even libraries use the query's
> time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${DAY} in your
> example. They use UTC timezone to do that. If you use localtime, like in
> your example (implicit) documents having 00:30 as timestamp for example
> will
> end up in the wrong index, and this will also depend on DST.
>
> When you search for them in kibana you might end up with wrong
> results.
>
> TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you
> use time-based indices.
>
> Maybe we should even add that to the default SCL.
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170512/5dd1e849/attachment.html>