Help! CentOS 7 ELK Stack from repos no index data.
So, I rebuilt my home stand alone system and only used the supported 3.8 and ES 2.4 current repos. Everything seems to be up and running with no complaints but I am not getting any data in Kibana, syslog-ng_* index and I can’t figure out if it’s something in syslog-ng.conf. Getting a bunch of stuff in /var/log/network.log from my cable modem but nothing is showing in discovery HELP! I’ve looked at this for hours and tried so many variants on destination d_es client-mode, all supported template options… Only thing weird I can see is this in the es logs. data=false [2016-09-08 00:00:01,977][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [] [2016-09-08 00:00:02,108][INFO ][cluster.routing.allocation] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[syslog-ng_2016.09.08][0], [syslog-ng_2016.09.08][0]] ...]). [2016-09-08 00:00:02,127][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] create_mapping [syslog-ng] [2016-09-08 00:15:28,886][INFO ][cluster.service ] [node-1] removed {{node-1}{kSmxkMoZQu6ZXvWWb70L5g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{kSmxkMoZQu6ZXvWWb70L5g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) [2016-09-08 00:15:39,358][INFO ][cluster.service ] [node-1] added {{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-join(join from node[{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}]) [2016-09-08 00:15:44,409][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] update_mapping [syslog-ng] [2016-09-08 00:31:14,268][INFO ][cluster.service ] [node-1] removed {{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) [2016-09-08 00:31:19,823][INFO ][cluster.service ] [node-1] added {{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-join(join from node[{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}]) [2016-09-08 00:38:29,163][INFO ][cluster.service ] [node-1] removed {{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-reposi... <https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html> https://www.elastic.co/guide/en/kibana/4.6/setup-repositories.html <https://www.elastic.co/guide/en/kibana/4.6/setup-repositories.html> https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-7/czani... <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-7/czanik-syslog-ng38-epel-7.repo> [root@meo syslog-ng]# cat /etc/syslog-ng/syslog-ng.conf @version:3.8 @include "scl.conf" @module mod-java options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); ts_format(iso); }; source s_netsyslog { udp(); tcp(); syslog(); }; source s_sys { system(); internal(); }; destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") port("9300") server("127.0.0.1") concurrent-requests("5") flush_limit("1") client-mode("transport") skip-cluster-health-check("yes") cluster("meo") custom_id("syslog-ng") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_es); }; log { source(s_sys); source(s_netsyslog); destination(d_netsyslog); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf"
On Thu, Sep 08, 2016 at 12:16:22AM -0400, Scot Needy wrote:
Everything seems to be up and running with no complaints but I am not getting any data in Kibana, syslog-ng_* index and I can’t figure out if it’s something in syslog-ng.conf.
You don't see anything in kibana, but do you see the index document count increasing? curl http://localhost:9200/_cat/indices If that's a yes, then there's many reasons why kibana could hide stuff from you, let's save that for the next thread :)
Thanks for the response I see the size growing but not the doc count. Flat log /var/log/network.log is getting new data every second health status index pri rep docs.count docs.deleted store.size pri.store.size [root@meo syslog-ng]# while true;do curl http://localhost:9200/_cat/indices;sleep 5;done yellow open .kibana 1 1 2 0 5.7kb 5.7kb yellow open syslog-ng_2016.09.08 5 1 1 3 12.7kb 12.7kb yellow open syslog-ng_2016.09.07 5 1 2 5 12.9kb 12.9kb yellow open .kibana 1 1 2 0 5.7kb 5.7kb yellow open syslog-ng_2016.09.08 5 1 1 5 12.7kb 12.7kb yellow open syslog-ng_2016.09.07 5 1 2 5 12.9kb 12.9kb yellow open .kibana 1 1 2 0 5.7kb 5.7kb yellow open syslog-ng_2016.09.08 5 1 1 1 12.1kb 12.1kb yellow open syslog-ng_2016.09.07 5 1 2 5 12.9kb 12.9kb yellow open .kibana 1 1 2 0 5.7kb 5.7kb yellow open syslog-ng_2016.09.08 5 1 1 0 12.1kb 12.1kb yellow open syslog-ng_2016.09.07 5 1 2 5 12.9kb 12.9kb
On Sep 8, 2016, at 3:25 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
curl http://localhost:9200/_cat/indices <http://localhost:9200/_cat/indices>
Hi Scot, On Thu, Sep 08, 2016 at 07:32:19AM -0400, Scot Needy wrote:
[root@meo syslog-ng]# while true;do curl http://localhost:9200/_cat/indices;sleep 5;done yellow open syslog-ng_2016.09.08 5 1 1 3 12.7kb 12.7kb
It may be that syslog-ng waits for the cluster to be green. The most common cause for that is that you configured elasticsearch for too many replicas that your cluster topology can handle. If you have only one node, make sure to reduce the number of replicas to 0 for every index [1] Cheers [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-upda...
Or try to set skip-cluster-health-check(yes) option in your elasticsearch destination: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... On Thu, Sep 8, 2016 at 2:40 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Scot,
On Thu, Sep 08, 2016 at 07:32:19AM -0400, Scot Needy wrote:
[root@meo syslog-ng]# while true;do curl http://localhost:9200/_cat/ indices;sleep 5;done yellow open syslog-ng_2016.09.08 5 1 1 3 12.7kb 12.7kb
It may be that syslog-ng waits for the cluster to be green. The most common cause for that is that you configured elasticsearch for too many replicas that your cluster topology can handle.
If you have only one node, make sure to reduce the number of replicas to 0 for every index [1]
Cheers
[1] https://www.elastic.co/guide/en/elasticsearch/reference/ current/indices-update-settings.html
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Good point. I saw that and thought this entry in syslog-ng.conf would address the health. It was already in there before I sent the msg. skip-cluster-health-check("yes”) That entry seems to have no effect. I’ll try setting it on the es side.
On Sep 8, 2016, at 8:49 AM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
Or try to set skip-cluster-health-check(yes) option in your elasticsearch destination: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-destination-elasticsearch2.html#elasticsearch2-option-elasticsearch2-skip-cluster-health-check>
On Thu, Sep 8, 2016 at 2:40 PM, Fabien Wernli <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote: Hi Scot,
On Thu, Sep 08, 2016 at 07:32:19AM -0400, Scot Needy wrote:
[root@meo syslog-ng]# while true;do curl http://localhost:9200/_cat/indices;sleep <http://localhost:9200/_cat/indices;sleep> 5;done yellow open syslog-ng_2016.09.08 5 1 1 3 12.7kb 12.7kb
It may be that syslog-ng waits for the cluster to be green. The most common cause for that is that you configured elasticsearch for too many replicas that your cluster topology can handle.
If you have only one node, make sure to reduce the number of replicas to 0 for every index [1]
Cheers
[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-upda... <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
Hi again, On Thu, Sep 08, 2016 at 12:16:22AM -0400, Scot Needy wrote:
destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") port("9300") server("127.0.0.1") concurrent-requests("5") flush_limit("1") client-mode("transport") skip-cluster-health-check("yes") cluster("meo") custom_id("syslog-ng") ^^^^^^^^^^^^^^^^^^^^^^ There's your problem: all documents will be assigned the literal "syslog-ng" as _id, so you're basically pushing all data overwriting the same document again and again :-)
sorry for not reading your config the first time
On Thu, Sep 08, 2016 at 03:43:17PM +0200, Fabien Wernli wrote:
custom_id("syslog-ng")
^^^^^^^^^^^^^^^^^^^^^^ There's your problem: all documents will be assigned the literal "syslog-ng" as _id, so you're basically pushing all data overwriting the same document again and again :-)
so the fix is simply to drop that option altogether
Thanks Fabien, that worked but not exactly sure why. I thought custom_id just added a tag to the document in ES.
On Sep 8, 2016, at 9:45 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Thu, Sep 08, 2016 at 03:43:17PM +0200, Fabien Wernli wrote:
custom_id("syslog-ng")
^^^^^^^^^^^^^^^^^^^^^^ There's your problem: all documents will be assigned the literal "syslog-ng" as _id, so you're basically pushing all data overwriting the same document again and again :-)
so the fix is simply to drop that option altogether
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Fabien Wernli
-
Fekete, Róbert
-
Scot Needy