[syslog-ng] Stupid E-S-K Question

[Radu Gheorghe]

Right, by default, if you send a string to a new field*, it will map
it as both text and keyword (in your case, HOST_FROM and
HOST_FROM.keyword respectively). Text fields are good for full-text
search (e.g. query for "web" will return "web-server01") and keyword
fields are used for sorting and aggregations (visualizations in
Kibana, those unique counts, for example, that will show
"web-server01" as a single token instead of "web" and "server01",
which is how text fields are analyzed by default).

* field that wasn't defined in a template:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Fri, Jan 27, 2017 at 9:41 AM, Scheidler, Balázs
<balazs.scheidler at balabit.com> wrote:
> So syslog-ng does send HOST in its output, so the problem is probably on the
> es side.
>
> On Jan 26, 2017 23:07, "Scot" <scotrn at gmail.com> wrote:
>>
>> On My test instance the only thing kibana shows are the "keyword" fields
>> like HOST_FROM.keyword but production has both HOST_FROM and
>> HOST_FROM.keyword.
>>
>> Perhaps from a previous es index or something ?
>>
>> Jan 26 16:54:19 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93
>> ret=-1
>> Jan 26 16:54:49 TheBarn Cannot find cache entry for mac 9c:e6:35:f2:cd:93
>> ret=-1
>> Output format applied
>> {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find
>> cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot
>> ","ISODATE":"2017-01-26T16:55:19-05:00","HOST_FROM":"192.168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan
>> 26 16:55:19"}
>> {"SOURCE":"s_net","PROGRAM":"Cannot","PRIORITY":"warning","MESSAGE":"find
>> cache entry for mac 9c:e6:35:f2:cd:93 ret=-1","LEGACY_MSGHDR":"Cannot
>> ","ISODATE":"2017-01-26T16:55:49-05:00","HOST_FROM":"192.168.1.1","HOST":"TheBarn","FACILITY":"user","DATE":"Jan
>> 26 16:55:49"}
>>
>>
>> On Wed, Jan 25, 2017 at 1:22 AM, Scheidler, Balázs
>> <balazs.scheidler at balabit.com> wrote:
>>>
>>> Can you post the format-json output so we can see if the HOST attribute
>>> is there?
>>>
>>> debug mode in syslog-ng should show that. Or alternatively you can use
>>> the same template to write to a throwaway logfile.
>>>
>>> On Jan 25, 2017 5:56 AM, "Scot" <scotrn at gmail.com> wrote:
>>>>
>>>> Elastic, Syslog-ng Kibana
>>>>
>>>> Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1
>>>>
>>>> I had a Kibana dashboard with a bar chart of unique count of systems
>>>> that had sent a syslog heartbeat. So I could see any missed heartbeats for
>>>> any host in the last 24 hours.
>>>>
>>>> Post upgrade of syslog-ng the host_from, host fields do not seem to come
>>>> into ES as usable fields because they are not indexed. So visualizations
>>>> "bar charts by unique 'host" is broken. Has anyone seen this?
>>>>
>>>>
>>>>                 client-mode("http")
>>>>                 index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
>>>>                 type("syslog") # Description: The type of the index. For
>>>> example, type("test")
>>>>                 template("$(format-json --scope rfc3164 --scope nv-pairs
>>>> --exclude R_DATE --key ISODATE)\n")
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>