question on elasticsearch & indexint structured data fields
Hi, I am testing elasticsearch with the 3.7.1 ose build on ubuntu 14.04 and have some questions regarding how to get elasticsearch & kibana to "see" the individual fields within a structured syslog message. I have tried a few different formats but all the >key>=<value> pairs appear within the MESSAGE part. For example: MESSAGE 2015-09-09T17:00:06.775 0055-inet-fw-node0 RT_FLOW - RT_FLOW_SESSION_CREATE_LS [junos@2636.1.1.1.2.35 logical-system-name="internetVR" source-address="143.115.190.50" source-port="42241" destination-address="70.39.233.137" destination-port="53" service-name="junos-dns-udp" nat-source-address="143.115.190.50" nat-source-port="42241" nat-destination-address="70.39.233.137" nat-destination-port="53" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="17" policy-name="Device-Zone-903" source-zone-name="dns-b2b" destination-zone-name="internet" session-id-32="80968105" username="N/A" roles="N/A" packet-incoming-interface="reth3.120" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN"] The "Available Fields" I see in kibana are: @timestamp tDATE tFACILITY tHOST tMESSAGE tPRIORITY tPROGRAM t_id t_index t_type I would like to be able to have each field recognized separately so that kibana could search for specific things like "source-address", etc. I thought these would be available under the .SDATA. set, but apparently I missed something. Is this possible (and I am just lacking understanding) or am I expecting too much? These are the last three tests I have used within the elasticsearch destination and they all essentially result in the same thing. #!# option( "message_template", "$(format-json --scope nv_pairs)\n") #!# option( "message_template", "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})\n") option( "message_template", "$(format-json --scope rfc5424 @timestamp=${ISODATE} --key .SDATA.* ) \n" ) Thanks for any help or guidance! Jim
Hi, I don't use Kibana regularly, but have some distant memories: in the upper right corner there is a "settings" icon. Once you click on it, "index pattern" will appear in the upper left corner with a pencil icon next to it. Click on it, and you will have an orange "reload field list" icon at the top of the screen. (this is with version 4.0) Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Thu, Sep 10, 2015 at 11:00 PM, <jrhendri@roadrunner.com> wrote:
Hi, I am testing elasticsearch with the 3.7.1 ose build on ubuntu 14.04 and have some questions regarding how to get elasticsearch & kibana to "see" the individual fields within a structured syslog message.
I have tried a few different formats but all the >key>=<value> pairs appear within the MESSAGE part. For example:
MESSAGE 2015-09-09T17:00:06.775 0055-inet-fw-node0 RT_FLOW - RT_FLOW_SESSION_CREATE_LS [junos@2636.1.1.1.2.35 logical-system-name="internetVR" source-address="143.115.190.50" source-port="42241" destination-address="70.39.233.137" destination-port="53" service-name="junos-dns-udp" nat-source-address="143.115.190.50" nat-source-port="42241" nat-destination-address="70.39.233.137" nat-destination-port="53" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="17" policy-name="Device-Zone-903" source-zone-name="dns-b2b" destination-zone-name="internet" session-id-32="80968105" username="N/A" roles="N/A" packet-incoming-interface="reth3.120" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN"]
The "Available Fields" I see in kibana are:
@timestamp tDATE tFACILITY tHOST tMESSAGE tPRIORITY tPROGRAM t_id t_index t_type
I would like to be able to have each field recognized separately so that kibana could search for specific things like "source-address", etc. I thought these would be available under the .SDATA. set, but apparently I missed something.
Is this possible (and I am just lacking understanding) or am I expecting too much?
These are the last three tests I have used within the elasticsearch destination and they all essentially result in the same thing.
#!# option( "message_template", "$(format-json --scope nv_pairs)\n") #!# option( "message_template", "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})\n") option( "message_template", "$(format-json --scope rfc5424 @timestamp=${ISODATE} --key .SDATA.* ) \n" )
Thanks for any help or guidance!
Jim
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Jim, On Thu, Sep 10, 2015 at 05:00:07PM -0400, jrhendri@roadrunner.com wrote:
I would like to be able to have each field recognized separately so that kibana could search for specific things like "source-address", etc. I thought these would be available under the .SDATA. set, but apparently I missed something.
Is this possible (and I am just lacking understanding) or am I expecting too much?
It's pretty much how it should work. As you can see from the 3.7 online guide, it's the `message_template` controls the fields which will be indexed in Elasticsearch. It's looking good in your example.
These are the last three tests I have used within the elasticsearch destination and they all essentially result in the same thing.
* Could you show us the full configuration? * Before looking into Kibana, you should use the elasticsearch API to list the fields e.g. by checking the mapping, dumping a document by id, or searching: (respectively) curl 0:9200/<index> curl 0:9200/<index>/<type>/<id> curl 0:9200/<index>/_search You're welcome to join #syslog-ng on freenode or #balabit/syslog_ng on gitter so we could move forward more quickly Cheers
Hi, I think this has to do with my testing using log messages that have already been written to a file. (i.e. I don't control the log server that receives the messages on the net - I have a sample file to parse) I have been doing some testing using various "no-parse" and "syslog-protocol" flags on the file source, as well as looking at what macros get which values on my end. So - long term, I would really want to be able to "replay" logs stored in flat files to (re-)ingest them into elasticsearch. That said - I think my current parsing issue may be due to the source being "cat >> file_source" Let me noodle on it over the weekend. There may be a simple answer here. Thanks again! Jim ---- Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Jim,
On Thu, Sep 10, 2015 at 05:00:07PM -0400, jrhendri@roadrunner.com wrote:
I would like to be able to have each field recognized separately so that kibana could search for specific things like "source-address", etc. I thought these would be available under the .SDATA. set, but apparently I missed something.
Is this possible (and I am just lacking understanding) or am I expecting too much?
It's pretty much how it should work. As you can see from the 3.7 online guide, it's the `message_template` controls the fields which will be indexed in Elasticsearch. It's looking good in your example.
These are the last three tests I have used within the elasticsearch destination and they all essentially result in the same thing.
* Could you show us the full configuration? * Before looking into Kibana, you should use the elasticsearch API to list the fields e.g. by checking the mapping, dumping a document by id, or searching: (respectively)
curl 0:9200/<index> curl 0:9200/<index>/<type>/<id> curl 0:9200/<index>/_search
You're welcome to join #syslog-ng on freenode or #balabit/syslog_ng on gitter so we could move forward more quickly
Cheers
participants (3)
-
Czanik, Péter
-
Fabien Wernli
-
jrhendri＠roadrunner.com