<div dir="ltr"><div><pre class="gmail-aLF-aPX-K0-aPE"><br></pre></div><div>Hi,</div><div><br></div><div>In your s_all source,you have multiple drivers referring the same file:</div><div></div><div><br></div><div><pre class="gmail-aLF-aPX-K0-aPE">source s_all {
internal();
unix-dgram("/run/systemd/journal/dev-log");
unix-stream("/run/systemd/journal/dev-log");
system();
file("/proc/kmsg" program-override("kernel"));
};<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">Both unix-dgram() and unix-stream() reference the same UNIX domain socket, which is just a filename on the filesystem. <br>So one of these drivers overwrite the other's socket. What's more the system() driver automatically finds out how to <br>collect local messages and in case systemd() support is enabled, it would possibly include yet another method for collecting the same.<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">What this would result in depends on whether you are using systemd (which I guess you do, as Ubuntu 18 has it), so your <br>configuration would effectively become (with systemd support):<br><br>source s_all {
internal();
unix-dgram("/run/systemd/journal/dev-log");
unix-stream("/run/systemd/journal/dev-log");
systemd-journal();
file("/proc/kmsg" program-override("kernel"));
};<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">And without systemd support:<br><br>source s_all {
internal();
unix-dgram("/run/systemd/journal/dev-log");
unix-stream("/run/systemd/journal/dev-log");
unix-dgram("/dev/log" so-rcvbuf(8192) flags(syslog-protocol));<br> file("/dev/kmsg" format(linux-kmsg));
file("/proc/kmsg" program-override("kernel"));
};<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">Note the differences in place of the original system() source. (the system source is an intelligent macro, it would expand to the <br>proper means of collecting local syslog data by looking at your environment/system type).<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">With all that said, without systemd, the effective source you'd be using is the 3rd unix-dgram instance (e.g. the one using /dev/log <br>as filename), simply because normally /dev/log is a symlink to /run/systemd/journal/dev-log, but the symlink would be overwritten <br>by the 3rd unix-dgram(), effectively making the first two completely moot.<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">With systemd however, you'd try to collect logs from systemd(), not overwriting /dev/log (however the original symlink may or may not <br>exist due to other attempts at fixing the issue, please confirm the symlink is in place).<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">If the symlink is there, only one of the unix-dgram() or unix-stream() drivers would effectively collect the messages and the last one<br></pre><pre class="gmail-aLF-aPX-K0-aPE">prevails. So you'd be effectively using unix-stream("/run/systemd/journal/dev-log");<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">And stream oriented transports, without framing do not support multiline messages, which is inline with the symptoms you are experiencing.<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">To fix this, please decide whether you want local logs through systemd (e.g. journal) or directly. And use only one of the means, <br>not all of them. If it's through journal, then just use the system source and remove the other two. If you want to skip journald, then <br>keep unix-dgram() either at the original location of /dev/log (which applications actually connect to) or through the symlink in /run/systemd/journal/dev-log.<br><br></pre><pre class="gmail-aLF-aPX-K0-aPE">This has become a bit lengthy, but I hope this helps.<br></pre></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 6, 2021 at 1:53 AM Jerry Scharf <<a href="mailto:scharf@lagunawayconsulting.com">scharf@lagunawayconsulting.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
We have been struggling with problems with multiple log messages being <br>
broken apart. Our apps can send random things to the logger including <br>
entire stack traces. We have a classifier that sends unknown lines to <br>
their own log file, so this makes a complete mess. This was with the <br>
distro apt package.<br>
<br>
So I went to building 3.25.1 by hand. If I build it with nothing on the <br>
configure line, it processes the multiline log messages just fine but if <br>
I build it with the ubuntu18 debian rules, it breaks the same way. I <br>
narrowed it down to if --eneable-systemd is set it breaks.<br>
<br>
Here are some relevant things.<br>
<br>
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"<br>
<br>
config cmd<br>
<br>
../configure --prefix=/usr/local --sysconfdir=/etc/syslog-ng <br>
--enable-dynamic-linking --enable-json --enable-manpages --enable-python <br>
--enable-java=no --enable-mongodb=no --enable-legacy-mongodb-options=no <br>
SOURCE_REVISION=3.25.1-ir1 --enable-systemd=no<br>
<br>
attaching the conf and the file classifier (60...)<br>
<br>
thanks,<br>
<br>
jerry<br>
<br>
<br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Bazsi</div>