[syslog-ng] Fields don't appear on kibana.

[Evan Rempel]

I'm relatively new to RFC5424 and was under the impression that all of the syslog-ng parsed values could/should be sent using the
structured data mechanism. This should apply to all macros, not just TAGS.

In general, what I am proposing is that any "work" done by syslog-ng on one computer, should be able to be maintained as the event gets passed
to another syslog-ng computer. The work should not be lost just because the payload has moved from one system to another.

I was hoping to use the syslog-ng pattern database to parse messages at the source, matching source ip addresses, user names etc.
and forward that vi RFC5424 to a syslog-ng system that would then feed elastic search, passing all of the parsed values to elastic search for indexing.
If that could not be done by RFC5424, I could use a raw JSON payload, and unwrap it at the receiving end, however, the source HOST, TAGS, DATE, TIME etc
are all lost in this case as well.

For the json parser, I think it would be good to have some kind of option for permitting core macros to be replaced/overwritten.
In the case of TAGS, which is a little bit special in the json object because it is converted to a string, rather than a json list, it should be appended to.

Evan.


On 09/03/2015 09:42 PM, Scheidler, Balázs wrote:
>
> you are right, it is a huge oversight. can you pls suggest an on wire format how this should work?
>
> -- 
> Bazsi
>
> This is a huge oversight. I have "complained" about this before. A JSON
> source (json parser) should append all of the tags from the JSON payload
> into the current set of TAGS.
>
> I'm not sure about syslog protocol (new RFC) if the TAGS is prepended
> with the .SDATA if the syslog parser will populate the TAGS. I would
> hope so.
>
> Evan.
>
> On 09/02/2015 12:00 AM, Fabien Wernli wrote:
> > Hi Balázs,
> >
> > On Wed, Sep 02, 2015 at 07:16:32AM +0200, Scheidler, Balázs wrote:
> >> The best solution to send dara over the wire between two Syslog-ng
> >> instances (e.g. the one getting the logs and the other storing them in
> >> elastic) is to use json to encode name-value pairs.
> > That's another way, indeed. What these have in common, though, is that there
> > is no way to transmit TAGS from one syslog-ng instance to another properly
> > (then use tags() filters on the remote end)
> >
> > ______________________________________________________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150904/64150496/attachment.htm