[syslog-ng] Elastic search data loading ?
Jim Hendrick
jrhendri at roadrunner.com
Thu Apr 14 14:33:54 CEST 2016
Just for grins, try querying ES directly with curl - it may be that
kibana isn't configured to show the right type//index/thingy...
had some issues like this with the old Kibana 4 a year or so ago -
haven't poked at it in a little while though.
jim
On 04/14/2016 01:37 AM, Scot Needy wrote:
>
> Hi,
>
> When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport”
>
> My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model.
> So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8 and ES2.
> I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup…
> But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below.
>
> NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution.
>
>
> destination d_es {
> elasticsearch2(
> index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
> type("syslog-ng") # Description: The type of the index. For example, type("test")
> template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})")
> #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
>
> port("9300")
> server("localhost")
> flush_limit("5000")
> client_mode("node")
> cluster("syslog-ng")
> custom_id("syslog")
> resource("/etc/elasticsearch/elasticsearch.yml")
> client_lib_dir("/usr/share/elasticsearch/lib")
> concurrent_requests("100")
> );
> };
>
> # sending logs to ES destination
> log {
> source(s_net);
> parser(pattern_db);
> destination(d_es);
> flags(flow-control);
> };
>
>
> /etc/elasticsearch/elasticsearch.yml
> cluster.name: syslog
> node.name: node-1
> path.home: /usr/share/elasticsearch
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
More information about the syslog-ng
mailing list