Just for grins, try querying ES directly with curl - it may be that kibana isn't configured to show the right type//index/thingy... had some issues like this with the old Kibana 4 a year or so ago - haven't poked at it in a little while though. jim On 04/14/2016 01:37 AM, Scot Needy wrote:
Hi,
When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport”
My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model. So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8 and ES2. I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup… But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below.
NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution.
destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
port("9300") server("localhost") flush_limit("5000") client_mode("node") cluster("syslog-ng") custom_id("syslog") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); };
# sending logs to ES destination log { source(s_net); parser(pattern_db); destination(d_es); flags(flow-control); };
/etc/elasticsearch/elasticsearch.yml cluster.name: syslog node.name: node-1 path.home: /usr/share/elasticsearch ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq