[syslog-ng] Elastic search data loading ?

[Scot Needy]

I think all the TCP port connections are correct  it’s just a configuration to get ES to store data.  

[root at loghost etc]# wget http://localhost:9200
--2016-04-14 09:37:48--  http://localhost:9200/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:9200... connected.
HTTP request sent, awaiting response... 200 OK
Length: 310 [application/json]
Saving to: ‘index.html’

100%[=========================================================================================================>] 310         --.-K/s   in 0s

2016-04-14 09:37:48 (20.9 MB/s) - ‘index.html’ saved [310/310]




> On Apr 14, 2016, at 8:33 AM, Jim Hendrick <jrhendri at roadrunner.com> wrote:
> 
> Just for grins, try querying ES directly with curl - it may be that 
> kibana isn't configured to show the right type//index/thingy...
> 
> had some issues like this with the old Kibana 4 a year or so ago - 
> haven't poked at it in a little while though.
> 
> jim
> 
> On 04/14/2016 01:37 AM, Scot Needy wrote:
>> 
>> Hi,
>> 
>>  When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport”
>> 
>> My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model.
>> So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8  and ES2.
>> I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup…
>> But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below.
>> 
>> NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution.
>> 
>> 
>> destination d_es {
>> 	elasticsearch2(
>> 		index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
>> 		type("syslog-ng") # Description: The type of the index. For example, type("test")
>> 		template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})")
>> 		#template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
>> 
>> 		port("9300")
>> 		server("localhost")
>> 		flush_limit("5000")
>> 		client_mode("node")
>> 		cluster("syslog-ng")
>> 		custom_id("syslog")
>> 		resource("/etc/elasticsearch/elasticsearch.yml")
>> 		client_lib_dir("/usr/share/elasticsearch/lib")
>> 		concurrent_requests("100")
>> 	);
>> };
>> 
>> # sending logs to ES destination
>> log {
>>   source(s_net);
>>   parser(pattern_db);
>>   destination(d_es);
>>   flags(flow-control);
>> };
>> 
>> 
>> /etc/elasticsearch/elasticsearch.yml
>> cluster.name: syslog
>> node.name: node-1
>> path.home: /usr/share/elasticsearch
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>