[syslog-ng] question on elasticsearch & indexint structured data fields

[jrhendri at roadrunner.com]

Hi,
  I am testing elasticsearch with the 3.7.1 ose build on ubuntu 14.04 and have some questions regarding how to get elasticsearch & kibana to "see" the individual fields within a structured syslog message.

I have tried a few different formats but all the >key>=<value> pairs appear within the MESSAGE part.
For example:

MESSAGE	2015-09-09T17:00:06.775 0055-inet-fw-node0 RT_FLOW - RT_FLOW_SESSION_CREATE_LS [junos at 2636.1.1.1.2.35 logical-system-name="internetVR" source-address="143.115.190.50" source-port="42241" destination-address="70.39.233.137" destination-port="53" service-name="junos-dns-udp" nat-source-address="143.115.190.50" nat-source-port="42241" nat-destination-address="70.39.233.137" nat-destination-port="53" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="17" policy-name="Device-Zone-903" source-zone-name="dns-b2b" destination-zone-name="internet" session-id-32="80968105" username="N/A" roles="N/A" packet-incoming-interface="reth3.120" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN"]

The "Available Fields" I see in kibana are:

@timestamp
tDATE
tFACILITY
tHOST
tMESSAGE
tPRIORITY
tPROGRAM
t_id
t_index
t_type

I would like to be able to have each field recognized separately so that kibana could search for specific things like "source-address", etc. I thought these would be available under the .SDATA. set, but apparently I missed something.

Is this possible (and I am just lacking understanding) or am I expecting too much?

These are the last three tests I have used within the elasticsearch destination and they all essentially result in the same thing.


#!#    option( "message_template", "$(format-json --scope nv_pairs)\n")
#!#    option( "message_template", "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})\n")
    option( "message_template", "$(format-json --scope rfc5424 @timestamp=${ISODATE} --key .SDATA.* ) \n" )

Thanks for any help or guidance!

Jim