Hi, In your s_all source,you have multiple drivers referring the same file: source s_all { internal(); unix-dgram("/run/systemd/journal/dev-log"); unix-stream("/run/systemd/journal/dev-log"); system(); file("/proc/kmsg" program-override("kernel")); }; Both unix-dgram() and unix-stream() reference the same UNIX domain socket, which is just a filename on the filesystem. So one of these drivers overwrite the other's socket. What's more the system() driver automatically finds out how to collect local messages and in case systemd() support is enabled, it would possibly include yet another method for collecting the same. What this would result in depends on whether you are using systemd (which I guess you do, as Ubuntu 18 has it), so your configuration would effectively become (with systemd support): source s_all { internal(); unix-dgram("/run/systemd/journal/dev-log"); unix-stream("/run/systemd/journal/dev-log"); systemd-journal(); file("/proc/kmsg" program-override("kernel")); }; And without systemd support: source s_all { internal(); unix-dgram("/run/systemd/journal/dev-log"); unix-stream("/run/systemd/journal/dev-log"); unix-dgram("/dev/log" so-rcvbuf(8192) flags(syslog-protocol)); file("/dev/kmsg" format(linux-kmsg)); file("/proc/kmsg" program-override("kernel")); }; Note the differences in place of the original system() source. (the system source is an intelligent macro, it would expand to the proper means of collecting local syslog data by looking at your environment/system type). With all that said, without systemd, the effective source you'd be using is the 3rd unix-dgram instance (e.g. the one using /dev/log as filename), simply because normally /dev/log is a symlink to /run/systemd/journal/dev-log, but the symlink would be overwritten by the 3rd unix-dgram(), effectively making the first two completely moot. With systemd however, you'd try to collect logs from systemd(), not overwriting /dev/log (however the original symlink may or may not exist due to other attempts at fixing the issue, please confirm the symlink is in place). If the symlink is there, only one of the unix-dgram() or unix-stream() drivers would effectively collect the messages and the last one prevails. So you'd be effectively using unix-stream("/run/systemd/journal/dev-log"); And stream oriented transports, without framing do not support multiline messages, which is inline with the symptoms you are experiencing. To fix this, please decide whether you want local logs through systemd (e.g. journal) or directly. And use only one of the means, not all of them. If it's through journal, then just use the system source and remove the other two. If you want to skip journald, then keep unix-dgram() either at the original location of /dev/log (which applications actually connect to) or through the symlink in /run/systemd/journal/dev-log. This has become a bit lengthy, but I hope this helps. On Sat, Feb 6, 2021 at 1:53 AM Jerry Scharf <scharf@lagunawayconsulting.com> wrote:
Hi,
We have been struggling with problems with multiple log messages being broken apart. Our apps can send random things to the logger including entire stack traces. We have a classifier that sends unknown lines to their own log file, so this makes a complete mess. This was with the distro apt package.
So I went to building 3.25.1 by hand. If I build it with nothing on the configure line, it processes the multiline log messages just fine but if I build it with the ubuntu18 debian rules, it breaks the same way. I narrowed it down to if --eneable-systemd is set it breaks.
Here are some relevant things.
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
config cmd
../configure --prefix=/usr/local --sysconfdir=/etc/syslog-ng --enable-dynamic-linking --enable-json --enable-manpages --enable-python --enable-java=no --enable-mongodb=no --enable-legacy-mongodb-options=no SOURCE_REVISION=3.25.1-ir1 --enable-systemd=no
attaching the conf and the file classifier (60...)
thanks,
jerry
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi