[syslog-ng] Convert json format to syslog rfc5424

[László Várady (lvarady)]

Hello,

> But, is it possible to do the opposite: read a json file and forward to a third server in syslog rfc5424 format?

Absolutely. json-parser() can be used to parse JSON objects into name-value pairs:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/72#TOPIC-1431181

You can then refer to the separated parts of the JSON message using the key of the JSON object as a macro.

RFC5424 can forward structured elements too, this part of the message is called "STRUCTURED-DATA".
If you add the ".SDATA." prefix to all important JSON fields, it will be automatically forwarded as RFC5424 structured-data.

For example:

log {
  source { file("/tmp/jsoninput" flags(no-parse)); };
  parser {
    json-parser(prefix(".SDATA.example at 32473.json."));
  };
  destination { syslog("host"); };
};

If you want to forward the original message (in JSON format), but with a proper RFC5424 header, you can just omit the json-parser() block.

Please note that RFC5424's SDATA has limitations: the name (key) of the sdata elements is limited to 32 characters:

SD-NAME         = 1*32PRINTUSASCII


Fortunately, syslog-ng won't truncate those names for you, but your collector may not be able to parse it back correctly:
https://github.com/syslog-ng/syslog-ng/issues/3197


When receiving JSON messages, you have to deal with message separation, because JSON parsing is a separate parser step.
For example, when reading from a file, you can use a basic file() source, if your JSON object is represented in a single line.
In case you receive JSON payload from network, I recommend using the syslog() source. This way, multi-line JSON data won't be a problem.

--
László Várady
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Carlos Lopez <clopmz at outlook.com>
Sent: Friday, May 22, 2020 8:59
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Convert json format to syslog rfc5424

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


Hi all,



Due to some issues with our logs collector server, I need to read some logs in json format and convert them to syslog rfc5424 and forward to our central syslog server.



Acctually, all information regarding this type of configuration located in syslog-ng’s web site is related to send json format as a destination. But, is it possible to do the opposite: read a json file and forward to a third server in syslog rfc5424 format? Yes, I will try to catch up only some specific fields, not all fields contained in the json soure file.



Any tips?



Regards,

C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200522/d77a09dc/attachment-0001.html>