I'm relatively new to RFC5424 and was under the impression that all of the syslog-ng parsed values could/should be sent using the structured data mechanism. This should apply to all macros, not just TAGS. In general, what I am proposing is that any "work" done by syslog-ng on one computer, should be able to be maintained as the event gets passed to another syslog-ng computer. The work should not be lost just because the payload has moved from one system to another. I was hoping to use the syslog-ng pattern database to parse messages at the source, matching source ip addresses, user names etc. and forward that vi RFC5424 to a syslog-ng system that would then feed elastic search, passing all of the parsed values to elastic search for indexing. If that could not be done by RFC5424, I could use a raw JSON payload, and unwrap it at the receiving end, however, the source HOST, TAGS, DATE, TIME etc are all lost in this case as well. For the json parser, I think it would be good to have some kind of option for permitting core macros to be replaced/overwritten. In the case of TAGS, which is a little bit special in the json object because it is converted to a string, rather than a json list, it should be appended to. Evan. On 09/03/2015 09:42 PM, Scheidler, Balázs wrote:
you are right, it is a huge oversight. can you pls suggest an on wire format how this should work?
-- Bazsi
This is a huge oversight. I have "complained" about this before. A JSON source (json parser) should append all of the tags from the JSON payload into the current set of TAGS.
I'm not sure about syslog protocol (new RFC) if the TAGS is prepended with the .SDATA if the syslog parser will populate the TAGS. I would hope so.
Evan.
On 09/02/2015 12:00 AM, Fabien Wernli wrote:
Hi Balázs,
On Wed, Sep 02, 2015 at 07:16:32AM +0200, Scheidler, Balázs wrote:
The best solution to send dara over the wire between two Syslog-ng instances (e.g. the one getting the logs and the other storing them in elastic) is to use json to encode name-value pairs. That's another way, indeed. What these have in common, though, is that there is no way to transmit TAGS from one syslog-ng instance to another properly (then use tags() filters on the remote end)
______________________________________________________________________________