[syslog-ng] Elastic search data loading ?

[Scot Needy]

This seems to be a problem with how Kibana is looking at the ES. 

from syslog-ng -F 
[2016-04-15T10:33:03.019083] org.syslog_ng.elasticsearch_v2.ElasticSearchDestination.createIndexRequest:95 - Outgoing log entry, json='{"PROGRAM":"asa11","PRIORITY":"warning","MESSAGE":"%ASA-4-313005: No matching connection for ICMP error message: icmp src outside:5.135.188.112 dst public:X.X.X.X (type 3, code 3) on outside interface.  Original IP payload: udp src X.X.X.X/3306 dst 5.135.188.112/3306.","ISODATE":"2016-04-15T10:33:03-04:00","HOST”:”X.X.X.X","FACILITY":"local5","timestamp":"2016-04-15T10:33:03-04:00"}’;

[2016-04-15T10:33:03.024982] org.syslog_ng.elasticsearch_v2.messageprocessor.ESSingleMessageProcessor.send:42 - Message inserted with id: syslog;

I can see the files growing in ES.  

[root at loghost kibana]# find /var/lib/elasticsearch/ -newer  /opt/syslog-ng/etc/syslog-ng.conf
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/_d.cfs
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/_d.si
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/_d.cfe
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/segments_c
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/translog
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/translog/translog.ckp
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/translog/translog-7.tlog
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/_6n5.cfs
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/_6n5.si
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/segments_4
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/_6n5.cfe
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/translog
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/translog/translog.ckp
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/translog/translog-4.tlog
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/_state
/var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/_state/state-3.st




> On Apr 14, 2016, at 11:03 AM, Scot Needy <scotrn at gmail.com> wrote:
> 
> [root at loghost etc]# curl http://localhost:9200/_cat/indices
> yellow open .kibana              1 1    2 0   7.6kb   7.6kb
> yellow open syslog-ng_2016.04.13 5 1 1110 1 383.5kb 383.5kb
> yellow open syslog-ng_2016.04.14 5 1    1 0  11.8kb  11.8kb
> 
> 
>> On Apr 14, 2016, at 10:47 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>> 
>> On Thu, Apr 14, 2016 at 09:41:42AM -0400, Scot Needy wrote:
>>> I think all the TCP port connections are correct  it’s just a configuration to get ES to store data.  
>> 
>> show the contents of the following please:
>> 
>>   wget http://localhost:9200/_cat/indices
>> 
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160415/71c60222/attachment.htm