[syslog-ng] problems with multiline log messages on ubuntu18

[Balazs Scheidler]

Hi,

In your s_all source,you have multiple drivers referring the same file:

source s_all {
	internal();
	unix-dgram("/run/systemd/journal/dev-log");
	unix-stream("/run/systemd/journal/dev-log");
	system();
	file("/proc/kmsg" program-override("kernel"));
};

Both unix-dgram() and unix-stream() reference the same UNIX domain
socket, which is just a filename on the filesystem.
So one of these drivers overwrite the other's socket. What's more the
system() driver automatically finds out how to
collect local messages and in case systemd() support is enabled, it
would possibly include yet another method for collecting the same.

What this would result in depends on whether you are using systemd
(which I guess you do, as Ubuntu 18 has it), so your
configuration would effectively become (with systemd support):

source s_all {
	internal();
	unix-dgram("/run/systemd/journal/dev-log");
	unix-stream("/run/systemd/journal/dev-log");
	systemd-journal();
	file("/proc/kmsg" program-override("kernel"));
};

And without systemd support:

source s_all {
	internal();
	unix-dgram("/run/systemd/journal/dev-log");
	unix-stream("/run/systemd/journal/dev-log");
	unix-dgram("/dev/log" so-rcvbuf(8192) flags(syslog-protocol));
        file("/dev/kmsg" format(linux-kmsg));
	file("/proc/kmsg" program-override("kernel"));
};

Note the differences in place of the original system() source. (the
system source is an intelligent macro, it would expand to the
proper means of collecting local syslog data by looking at your
environment/system type).

With all that said, without systemd, the effective source you'd be
using is the 3rd unix-dgram instance (e.g. the one using /dev/log
as filename), simply because normally /dev/log is a symlink to
/run/systemd/journal/dev-log, but the symlink would be overwritten
by the 3rd unix-dgram(), effectively making the first two completely moot.

With systemd however, you'd try to collect logs from systemd(), not
overwriting /dev/log (however the original symlink may or may not
exist due to other attempts at fixing the issue, please confirm the
symlink is in place).

If the symlink is there, only one of the unix-dgram() or unix-stream()
drivers would effectively collect the messages and the last one

prevails. So you'd be effectively using
unix-stream("/run/systemd/journal/dev-log");

And stream oriented transports, without framing do not support
multiline messages, which is inline with the symptoms you are
experiencing.

To fix this, please decide whether you want local logs through systemd
(e.g. journal) or directly. And use only one of the means,
not all of them. If it's through journal, then just use the system
source and remove the other two. If you want to skip journald, then
keep unix-dgram() either at the original location  of /dev/log (which
applications actually connect to) or through the symlink in
/run/systemd/journal/dev-log.

This has become a bit lengthy, but I hope this helps.


On Sat, Feb 6, 2021 at 1:53 AM Jerry Scharf <scharf at lagunawayconsulting.com>
wrote:

> Hi,
>
> We have been struggling with problems with multiple log messages being
> broken apart. Our apps can send random things to the logger including
> entire stack traces. We have a classifier that sends unknown lines to
> their own log file, so this makes a complete mess. This was with the
> distro apt package.
>
> So I went to building 3.25.1 by hand. If I build it with nothing on the
> configure line, it processes the multiline log messages just fine but if
> I build it with the ubuntu18 debian rules, it breaks the same way. I
> narrowed it down to if --eneable-systemd is set it breaks.
>
> Here are some relevant things.
>
> DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
>
> config cmd
>
> ../configure --prefix=/usr/local --sysconfdir=/etc/syslog-ng
> --enable-dynamic-linking --enable-json --enable-manpages --enable-python
> --enable-java=no --enable-mongodb=no --enable-legacy-mongodb-options=no
> SOURCE_REVISION=3.25.1-ir1 --enable-systemd=no
>
> attaching the conf and the file classifier (60...)
>
> thanks,
>
> jerry
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

-- 
Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210206/195ed37f/attachment-0001.html>