Stupid E-S-K Question
*E*lastic, *S*yslog-ng *K*ibana Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1 I had a Kibana dashboard with a bar chart of unique count of systems that had sent a syslog heartbeat. So I could see any missed heartbeats for any host in the last 24 hours. Post upgrade of syslog-ng the host_from, host fields do not seem to come into ES as usable fields because they are not indexed. So visualizations "bar charts by unique 'host" is broken. Has anyone seen this? client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
Can you post the format-json output so we can see if the HOST attribute is there? debug mode in syslog-ng should show that. Or alternatively you can use the same template to write to a throwaway logfile. On Jan 25, 2017 5:56 AM, "Scot" <scotrn@gmail.com> wrote:
*E*lastic, *S*yslog-ng *K*ibana
Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1
I had a Kibana dashboard with a bar chart of unique count of systems that had sent a syslog heartbeat. So I could see any missed heartbeats for any host in the last 24 hours.
Post upgrade of syslog-ng the host_from, host fields do not seem to come into ES as usable fields because they are not indexed. So visualizations "bar charts by unique 'host" is broken. Has anyone seen this?
client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On My test instance the only thing kibana shows are the "keyword" fields like HOST_FROM.keyword but production has both HOST_FROM and HOST_FROM.keyword. Perhaps from a previous es index or something ? Jan 26 16:54:19 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1 Jan 26 16:54:49 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1 *Output format applied * {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot ","ISODATE":"2017-01-26T16:55:19-05:00","HOST_FROM":"192.168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan 26 16:55:19"} {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot ","ISODATE":"2017-01-26T16:55:49-05:00","HOST_FROM":"192.168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan 26 16:55:49"} On Wed, Jan 25, 2017 at 1:22 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you post the format-json output so we can see if the HOST attribute is there?
debug mode in syslog-ng should show that. Or alternatively you can use the same template to write to a throwaway logfile.
On Jan 25, 2017 5:56 AM, "Scot" <scotrn@gmail.com> wrote:
*E*lastic, *S*yslog-ng *K*ibana
Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1
I had a Kibana dashboard with a bar chart of unique count of systems that had sent a syslog heartbeat. So I could see any missed heartbeats for any host in the last 24 hours.
Post upgrade of syslog-ng the host_from, host fields do not seem to come into ES as usable fields because they are not indexed. So visualizations "bar charts by unique 'host" is broken. Has anyone seen this?
client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
So syslog-ng does send HOST in its output, so the problem is probably on the es side. On Jan 26, 2017 23:07, "Scot" <scotrn@gmail.com> wrote:
On My test instance the only thing kibana shows are the "keyword" fields like HOST_FROM.keyword but production has both HOST_FROM and HOST_FROM.keyword.
Perhaps from a previous es index or something ?
Jan 26 16:54:19 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1 Jan 26 16:54:49 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1 *Output format applied * {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot ","ISODATE":"2017-01-26T16:55:19-05:00","HOST_FROM":"192. 168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan 26 16:55:19"} {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot ","ISODATE":"2017-01-26T16:55:49-05:00","HOST_FROM":"192. 168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan 26 16:55:49"}
On Wed, Jan 25, 2017 at 1:22 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you post the format-json output so we can see if the HOST attribute is there?
debug mode in syslog-ng should show that. Or alternatively you can use the same template to write to a throwaway logfile.
On Jan 25, 2017 5:56 AM, "Scot" <scotrn@gmail.com> wrote:
*E*lastic, *S*yslog-ng *K*ibana
Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1
I had a Kibana dashboard with a bar chart of unique count of systems that had sent a syslog heartbeat. So I could see any missed heartbeats for any host in the last 24 hours.
Post upgrade of syslog-ng the host_from, host fields do not seem to come into ES as usable fields because they are not indexed. So visualizations "bar charts by unique 'host" is broken. Has anyone seen this?
client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Right, by default, if you send a string to a new field*, it will map it as both text and keyword (in your case, HOST_FROM and HOST_FROM.keyword respectively). Text fields are good for full-text search (e.g. query for "web" will return "web-server01") and keyword fields are used for sorting and aggregations (visualizations in Kibana, those unique counts, for example, that will show "web-server01" as a single token instead of "web" and "server01", which is how text fields are analyzed by default). * field that wasn't defined in a template: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-temp... -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Jan 27, 2017 at 9:41 AM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
So syslog-ng does send HOST in its output, so the problem is probably on the es side.
On Jan 26, 2017 23:07, "Scot" <scotrn@gmail.com> wrote:
On My test instance the only thing kibana shows are the "keyword" fields like HOST_FROM.keyword but production has both HOST_FROM and HOST_FROM.keyword.
Perhaps from a previous es index or something ?
Jan 26 16:54:19 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1 Jan 26 16:54:49 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1 Output format applied {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot ","ISODATE":"2017-01-26T16:55:19-05:00","HOST_FROM":"192.168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan 26 16:55:19"} {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot ","ISODATE":"2017-01-26T16:55:49-05:00","HOST_FROM":"192.168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan 26 16:55:49"}
On Wed, Jan 25, 2017 at 1:22 AM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
Can you post the format-json output so we can see if the HOST attribute is there?
debug mode in syslog-ng should show that. Or alternatively you can use the same template to write to a throwaway logfile.
On Jan 25, 2017 5:56 AM, "Scot" <scotrn@gmail.com> wrote:
Elastic, Syslog-ng Kibana
Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1
I had a Kibana dashboard with a bar chart of unique count of systems that had sent a syslog heartbeat. So I could see any missed heartbeats for any host in the last 24 hours.
Post upgrade of syslog-ng the host_from, host fields do not seem to come into ES as usable fields because they are not indexed. So visualizations "bar charts by unique 'host" is broken. Has anyone seen this?
client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Radu Gheorghe
-
Scheidler, Balázs
-
Scot