<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I think the simplest approach would be if you feed a properly formatted json into the source (just add {} brackets to _PRINT_DEBUG). Use json-parser to turn the input into name-value pairs. Then print everything with the format-json template function.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I simulated the json input with example-msg generator for sake of simplicity, but I think similar config could work for your source too.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: Consolas, Courier, monospace;">@version: 3.26</span><span><br>
</span>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace;">log {</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace;"> source { example-msg-generator(num(1) template("{\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Check pstore partition..\"}")); };</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace;"> parser { json-parser(); };</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace;"> destination { file(/dev/stdout template("$(format-json --key category,level,message timestamp=${ISODATE})\n")); };</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace;">};</span><br>
</div>
<span></span></div>
<br>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This outputs:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: Consolas, Courier, monospace;">{"timestamp":"2020-04-24T12:37:58+02:00","message":"Check pstore partition..","level":"INFO","category":"BOOT"}</span><span><br>
</span><span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt">
Br,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt">
Antal</div>
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Arthur Lambert <lambertarthur22@gmail.com><br>
<b>Sent:</b> Tuesday, April 21, 2020 11:56<br>
<b>To:</b> syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] syslog-ng, json and handle escaping properly</font>
<div> </div>
</div>
<div>
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div dir="ltr">Hello,<br>
<br>
I tried to migrate from syslog to Syslog-ng. I am trying to deploy it on an embedded target<br>
with buildroot. At the same time, I also tried to switch log format to JSON.<br>
<br>
In my c code, I use Syslog function and init with the openlog function:<br>
openlog("middleware", 0, LOG_LOCAL1);<br>
<br>
I have some macro to generate the json logs:<br>
#define _PRINT_DEBUG(M, ...) \<br>
do { \<br>
syslog(LOG_DEBUG, "\"category\":\"%s\",\"level\":\"DEBUG\",\"message\":\"" M "%s\"}\n", __DIR__, __VA_ARGS__); \<br>
} while (0)<br>
<br>
I define a syslog-ng conf file:<br>
<br>
@version: 3.2X<br>
<br>
filter f_middleware {<br>
facility("local1");<br>
};<br>
<br>
template t_json_filetemplate {<br>
template("{\"timestamp\":\"${ISODATE}\",${MESSAGE}\n");<br>
};<br>
<br>
source s_kernel {<br>
file("/proc/kmsg" program_override("kernel"));<br>
};<br>
<br>
source s_middleware {<br>
unix-stream("/dev/log");<br>
};<br>
<br>
destination d_kernel {<br>
file("/data/logs/kern.log");<br>
};<br>
<br>
destination d_middleware {<br>
file("/data/logs/middleware.log", template(t_json_filetemplate));<br>
};<br>
<br>
log {<br>
source(s_kernel);<br>
destination(d_kernel);<br>
};<br>
<br>
log {<br>
source(s_middleware);<br>
filter(f_middleware);<br>
destination(d_middleware);<br>
};<br>
<br>
So basically I want to redirect all my log from my code in /data/logs/middleware.log. I also want to redirect all kernel logs<br>
in /data/logs/kern.log<br>
<br>
I have two issues. The first one is escaping.<br>
If I want to print a JSON file the double quote will be not escaped. My first solution was to use escape feature from Syslog-ng:<br>
destination d_middleware {<br>
file("/data/logs/middleware.log", template("{\"timestamp\":\"${ISODATE}\",${MESSAGE}\n") template-escape(yes)); <br>
<br>
};<br>
<br>
Problem is that now everything is escaped and not only the message, some example:<br>
{"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Check pstore partition..\"}<br>
{"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Start hardware watchdog daemon...\"}<br>
{"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Prepare asound configuration...\"}<br>
{"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Middleware is about to start...\"}<br>
{"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Load sdma firmware..\"}<br>
{"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Watchdog is about to start...\"}<br>
{"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Start mpu watchdog..\"}<br>
<br>
As you can see, everything is escaped because for Syslog-ng, everything is the message and not only the final part. I feel like I am completely<br>
using Syslog-ng in the wrong way right? How can I fix that? Do I have to handle escaping by myself manually in my middleware?<br>
Do I have to pre-process all logs manually by myself? It seems quite awful.<br>
<br>
I have a second issue which is probably related? When I tried to print the content of a file, the last '}' is truncated. The reason seems to be related<br>
to the presence of \n. I can reproduce the issue by just calling my PRINT_DEBUG macro as follow:<br>
<br>
PRINT_DEBUG ("YO YO YO KO\n");<br>
PRINT_DEBUG ("YO YO YO OK");<br>
<br>
The result:<br>
<br>
{"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"MPU_CORE\",\"level\":\"DEBUG\",\"message\":\"YO YO YO KO<br>
{"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"MPU_CORE\",\"level\":\"DEBUG\",\"message\":\"YO YO YO OK\"}<br>
<br>
My syslog version/module information :<br>
<br>
# syslog-ng -V -s<br>
syslog-ng 3 (3.26.1)<br>
Config version: 3.22<br>
Installer-Version: 3.26.1<br>
Revision:<br>
Compile-Date: Apr 10 2020 16:41:46<br>
Module-Directory: /usr/lib/syslog-ng<br>
Module-Path: /usr/lib/syslog-ng<br>
Include-Path: /usr/share/syslog-ng/include<br>
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,dbparser,disk-buffer,examples,graphite,hook-commands,ht<br>
tp,kvformat,linux-kmsg-format,map-value-pairs,mod-python,pseudofile,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,timestamp,xml<br>
Enable-Debug: off<br>
Enable-GProf: off<br>
Enable-Memtrace: off<br>
Enable-IPv6: on<br>
Enable-Spoof-Source: off<br>
Enable-TCP-Wrapper: off<br>
Enable-Linux-Caps: off<br>
Enable-Systemd: off<br>
<br>
--<br>
- Arthur LAMBERT</div>
</div>
</div>
</body>
</html>