Elastic search data loading ?
Hi, When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport” My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model. So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8 and ES2. I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup… But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below. NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution. destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") port("9300") server("localhost") flush_limit("5000") client_mode("node") cluster("syslog-ng") custom_id("syslog") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); }; # sending logs to ES destination log { source(s_net); parser(pattern_db); destination(d_es); flags(flow-control); }; /etc/elasticsearch/elasticsearch.yml cluster.name: syslog node.name: node-1 path.home: /usr/share/elasticsearch
Just for grins, try querying ES directly with curl - it may be that kibana isn't configured to show the right type//index/thingy... had some issues like this with the old Kibana 4 a year or so ago - haven't poked at it in a little while though. jim On 04/14/2016 01:37 AM, Scot Needy wrote:
Hi,
When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport”
My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model. So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8 and ES2. I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup… But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below.
NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution.
destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
port("9300") server("localhost") flush_limit("5000") client_mode("node") cluster("syslog-ng") custom_id("syslog") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); };
# sending logs to ES destination log { source(s_net); parser(pattern_db); destination(d_es); flags(flow-control); };
/etc/elasticsearch/elasticsearch.yml cluster.name: syslog node.name: node-1 path.home: /usr/share/elasticsearch ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I think all the TCP port connections are correct it’s just a configuration to get ES to store data. [root@loghost etc]# wget http://localhost:9200 --2016-04-14 09:37:48-- http://localhost:9200/ Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:9200... connected. HTTP request sent, awaiting response... 200 OK Length: 310 [application/json] Saving to: ‘index.html’ 100%[=========================================================================================================>] 310 --.-K/s in 0s 2016-04-14 09:37:48 (20.9 MB/s) - ‘index.html’ saved [310/310]
On Apr 14, 2016, at 8:33 AM, Jim Hendrick <jrhendri@roadrunner.com> wrote:
Just for grins, try querying ES directly with curl - it may be that kibana isn't configured to show the right type//index/thingy...
had some issues like this with the old Kibana 4 a year or so ago - haven't poked at it in a little while though.
jim
On 04/14/2016 01:37 AM, Scot Needy wrote:
Hi,
When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport”
My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model. So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8 and ES2. I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup… But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below.
NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution.
destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
port("9300") server("localhost") flush_limit("5000") client_mode("node") cluster("syslog-ng") custom_id("syslog") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); };
# sending logs to ES destination log { source(s_net); parser(pattern_db); destination(d_es); flags(flow-control); };
/etc/elasticsearch/elasticsearch.yml cluster.name: syslog node.name: node-1 path.home: /usr/share/elasticsearch ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Thu, Apr 14, 2016 at 09:41:42AM -0400, Scot Needy wrote:
I think all the TCP port connections are correct it’s just a configuration to get ES to store data.
show the contents of the following please: wget http://localhost:9200/_cat/indices
[root@loghost etc]# curl http://localhost:9200/_cat/indices yellow open .kibana 1 1 2 0 7.6kb 7.6kb yellow open syslog-ng_2016.04.13 5 1 1110 1 383.5kb 383.5kb yellow open syslog-ng_2016.04.14 5 1 1 0 11.8kb 11.8kb
On Apr 14, 2016, at 10:47 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Thu, Apr 14, 2016 at 09:41:42AM -0400, Scot Needy wrote:
I think all the TCP port connections are correct it’s just a configuration to get ES to store data.
show the contents of the following please:
wget http://localhost:9200/_cat/indices
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
This seems to be a problem with how Kibana is looking at the ES. from syslog-ng -F [2016-04-15T10:33:03.019083] org.syslog_ng.elasticsearch_v2.ElasticSearchDestination.createIndexRequest:95 - Outgoing log entry, json='{"PROGRAM":"asa11","PRIORITY":"warning","MESSAGE":"%ASA-4-313005: No matching connection for ICMP error message: icmp src outside:5.135.188.112 dst public:X.X.X.X (type 3, code 3) on outside interface. Original IP payload: udp src X.X.X.X/3306 dst 5.135.188.112/3306.","ISODATE":"2016-04-15T10:33:03-04:00","HOST”:”X.X.X.X","FACILITY":"local5","timestamp":"2016-04-15T10:33:03-04:00"}’; [2016-04-15T10:33:03.024982] org.syslog_ng.elasticsearch_v2.messageprocessor.ESSingleMessageProcessor.send:42 - Message inserted with id: syslog; I can see the files growing in ES. [root@loghost kibana]# find /var/lib/elasticsearch/ -newer /opt/syslog-ng/etc/syslog-ng.conf /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/_d.cfs /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/_d.si /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/_d.cfe /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/index/segments_c /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/translog /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/translog/translog.ckp /var/lib/elasticsearch/syslog-ng/nodes/0/indices/.kibana/0/translog/translog-7.tlog /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/_6n5.cfs /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/_6n5.si /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/segments_4 /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/index/_6n5.cfe /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/translog /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/translog/translog.ckp /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/3/translog/translog-4.tlog /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/_state /var/lib/elasticsearch/syslog-ng/nodes/0/indices/syslog-ng_2016.04.15/_state/state-3.st
On Apr 14, 2016, at 11:03 AM, Scot Needy <scotrn@gmail.com> wrote:
[root@loghost etc]# curl http://localhost:9200/_cat/indices yellow open .kibana 1 1 2 0 7.6kb 7.6kb yellow open syslog-ng_2016.04.13 5 1 1110 1 383.5kb 383.5kb yellow open syslog-ng_2016.04.14 5 1 1 0 11.8kb 11.8kb
On Apr 14, 2016, at 10:47 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Thu, Apr 14, 2016 at 09:41:42AM -0400, Scot Needy wrote:
I think all the TCP port connections are correct it’s just a configuration to get ES to store data.
show the contents of the following please:
wget http://localhost:9200/_cat/indices
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, Apr 15, 2016 at 10:38:48AM -0400, Scot Needy wrote:
This seems to be a problem with how Kibana is looking at the ES.
yes, and as such it's an ES/kibana issue. I'd be happy to assist you further through another channel, e.g. on IRC or through email. The next steps for you are probably: * set the timestamp in kibana * add a default field in your index mapping
I don’t see you on IRC. I think I am making progress but not sure how. Shutdown ES Kibana and syslog-ng. rm-rf /var/lib/elasticsearch/* Started ES Started syslog-ng. Startred Kibaba. I see logs.. ! Must be a simpler way. Now I need to tune GeoIP and pattern_db.
On Apr 15, 2016, at 10:46 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Fri, Apr 15, 2016 at 10:38:48AM -0400, Scot Needy wrote:
This seems to be a problem with how Kibana is looking at the ES.
yes, and as such it's an ES/kibana issue. I'd be happy to assist you further through another channel, e.g. on IRC or through email.
The next steps for you are probably:
* set the timestamp in kibana * add a default field in your index mapping
participants (3)
-
Fabien Wernli
-
Jim Hendrick
-
Scot Needy