[syslog-ng] Elastic search data loading ?

[Jim Hendrick]

Just for grins, try querying ES directly with curl - it may be that 
kibana isn't configured to show the right type//index/thingy...

had some issues like this with the old Kibana 4 a year or so ago - 
haven't poked at it in a little while though.

jim

On 04/14/2016 01:37 AM, Scot Needy wrote:
>
> Hi,
>
>   When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport”
>
> My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model.
> So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8  and ES2.
> I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup…
> But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below.
>
> NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution.
>
>
> destination d_es {
> 	elasticsearch2(
> 		index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
> 		type("syslog-ng") # Description: The type of the index. For example, type("test")
> 		template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})")
> 		#template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
>
> 		port("9300")
> 		server("localhost")
> 		flush_limit("5000")
> 		client_mode("node")
> 		cluster("syslog-ng")
> 		custom_id("syslog")
> 		resource("/etc/elasticsearch/elasticsearch.yml")
> 		client_lib_dir("/usr/share/elasticsearch/lib")
> 		concurrent_requests("100")
> 	);
> };
>
> # sending logs to ES destination
> log {
>    source(s_net);
>    parser(pattern_db);
>    destination(d_es);
>    flags(flow-control);
> };
>
>
> /etc/elasticsearch/elasticsearch.yml
> cluster.name: syslog
> node.name: node-1
> path.home: /usr/share/elasticsearch
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>