[syslog-ng] question on elasticsearch & indexint structured data fields

[Czanik, Péter]

Hi,

I don't use Kibana regularly, but have some distant memories: in the
upper right corner there is a "settings" icon. Once you click on it,
"index pattern" will appear in the upper left corner with a pencil
icon next to it. Click on it, and you will have an orange "reload
field list" icon at the top of the screen. (this is with version 4.0)

Bye,
Peter Czanik (CzP) <peter.czanik at balabit.com>
BalaBit IT Security / syslog-ng upstream
http://czanik.blogs.balabit.com/
https://twitter.com/PCzanik


On Thu, Sep 10, 2015 at 11:00 PM,  <jrhendri at roadrunner.com> wrote:
> Hi,
>   I am testing elasticsearch with the 3.7.1 ose build on ubuntu 14.04 and have some questions regarding how to get elasticsearch & kibana to "see" the individual fields within a structured syslog message.
>
> I have tried a few different formats but all the >key>=<value> pairs appear within the MESSAGE part.
> For example:
>
> MESSAGE 2015-09-09T17:00:06.775 0055-inet-fw-node0 RT_FLOW - RT_FLOW_SESSION_CREATE_LS [junos at 2636.1.1.1.2.35 logical-system-name="internetVR" source-address="143.115.190.50" source-port="42241" destination-address="70.39.233.137" destination-port="53" service-name="junos-dns-udp" nat-source-address="143.115.190.50" nat-source-port="42241" nat-destination-address="70.39.233.137" nat-destination-port="53" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="17" policy-name="Device-Zone-903" source-zone-name="dns-b2b" destination-zone-name="internet" session-id-32="80968105" username="N/A" roles="N/A" packet-incoming-interface="reth3.120" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN"]
>
> The "Available Fields" I see in kibana are:
>
> @timestamp
> tDATE
> tFACILITY
> tHOST
> tMESSAGE
> tPRIORITY
> tPROGRAM
> t_id
> t_index
> t_type
>
> I would like to be able to have each field recognized separately so that kibana could search for specific things like "source-address", etc. I thought these would be available under the .SDATA. set, but apparently I missed something.
>
> Is this possible (and I am just lacking understanding) or am I expecting too much?
>
> These are the last three tests I have used within the elasticsearch destination and they all essentially result in the same thing.
>
>
> #!#    option( "message_template", "$(format-json --scope nv_pairs)\n")
> #!#    option( "message_template", "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})\n")
>     option( "message_template", "$(format-json --scope rfc5424 @timestamp=${ISODATE} --key .SDATA.* ) \n" )
>
> Thanks for any help or guidance!
>
> Jim
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>