rsyslog client produces "Error processing log message"
Hello, we have a centralised log server running syslog-ng 3.1 OSE on Debian 6.0. On the client side, we were using syslog-ng but now I'd like to use rsyslog instead (for several reasons). Transport should be TLS-encrypted TCP. I have set up a connection between the two, but apparently syslog-ng fails to parse the log messages sent by rsyslog. Every log line goes like this: Nov 6 11:15:31 admin2-desktop syslog-ng[1578]: Error processing log message: <13>Nov 6 11:15:31 admin2-desktop ah: Test4 Does anyone have an idea what to configure with either rsyslog or syslog-ng so the two understand each other? Relevant server side config: source s_all { syslog(ip(172.16.x.x) port(6514) max_connections(50) tls( key_file("/etc/ssl/private/xxx.pem") cert_file("/etc/ssl/private/xxx.pem") ca_dir("/etc/ssl/certs") peer_verify(optional-untrusted) ) ); }; Relevant client side config: $DefaultNetstreamDriverCAFile /etc/ssl/certs/xxx.pem $DefaultNetstreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated *.* @@(o)mail:6514;RSYSLOG_TraditionalForwardFormat # (o) seems to be neccessary or syslog-ng will not even accept the connection; RSYSLOG_TraditionalForwardFormat apparently doesn't matter Thanks, Andreas
Andreas Heinlein <aheinlein@gmx.com> writes:
we have a centralised log server running syslog-ng 3.1 OSE on Debian 6.0. On the client side, we were using syslog-ng but now I'd like to use rsyslog instead (for several reasons).
Independently of the issue below, I'd love to hear the reasons (either on-list, or in private).
Transport should be TLS-encrypted TCP. I have set up a connection between the two, but apparently syslog-ng fails to parse the log messages sent by rsyslog. Every log line goes like this:
Nov 6 11:15:31 admin2-desktop syslog-ng[1578]: Error processing log message: <13>Nov 6 11:15:31 admin2-desktop ah: Test4
Does anyone have an idea what to configure with either rsyslog or syslog-ng so the two understand each other?
Relevant server side config: source s_all { syslog(ip(172.16.x.x) port(6514) max_connections(50) tls( ^^^^^^
This is the issue. You're telling syslog-ng to expect the new syslog protocol, but later in the rsyslog.conf, you don't seem to be telling it to send that version, so it will use the legacy BSD format instead. You have two options: either use tcp() on the syslog-ng side, or ask rsyslog to forward messages according to the new syslog protocol (however it may call it, it's RFC5424 by the way, while RFC3164 is the legacy BSD format). -- |8]
Gergely Nagy <algernon <at> balabit.hu> writes:
Andreas Heinlein <aheinlein <at> gmx.com> writes:
we have a centralised log server running syslog-ng 3.1 OSE on Debian 6.0. On the client side, we were using syslog-ng but now I'd like to use rsyslog instead (for several reasons).
Independently of the issue below, I'd love to hear the reasons (either on-list, or in private).
The main reason is that syslog-ng randomly goes to 100% CPU on the clients (Ubuntu 12.04, previously we used 10.04 and syslog-ng which worked just fine), and I've been trying for several days now to figure out why. So I've decided to try and switch to rsyslog.
Transport should be TLS-encrypted TCP. I have set up a connection between the two, but apparently syslog-ng fails to parse the log messages sent by rsyslog. Every log line goes like this:
Nov 6 11:15:31 admin2-desktop syslog-ng[1578]: Error processing log message: <13>Nov 6 11:15:31 admin2-desktop ah: Test4
Does anyone have an idea what to configure with either rsyslog or syslog-ng so the two understand each other?
Relevant server side config: source s_all { syslog(ip(172.16.x.x) port(6514) max_connections(50) tls( ^^^^^^
This is the issue. You're telling syslog-ng to expect the new syslog protocol, but later in the rsyslog.conf, you don't seem to be telling it to send that version, so it will use the legacy BSD format instead.
You have two options: either use tcp() on the syslog-ng side, or ask rsyslog to forward messages according to the new syslog protocol (however it may call it, it's RFC5424 by the way, while RFC3164 is the legacy BSD format).
I suspected that, but I can't figure out what to configure in rsyslog.conf then. I thought that it would use the new RFC5424 when *not* using RSYSLOG_TraditionalForwardFormat, but maybe it doesn't. I should probably ask on the rsyslog mailing list then. The other way round would break existing syslog-ng clients, that is no option. Thanks for your effort! Andreas
Andreas Heinlein <aheinlein@gmx.com> writes: xx>> > we have a centralised log server running syslog-ng 3.1 OSE on Debian
6.0. On the client side, we were using syslog-ng but now I'd like to use rsyslog instead (for several reasons).
Independently of the issue below, I'd love to hear the reasons (either on-list, or in private).
The main reason is that syslog-ng randomly goes to 100% CPU on the clients (Ubuntu 12.04, previously we used 10.04 and syslog-ng which worked just fine), and I've been trying for several days now to figure out why. So I've decided to try and switch to rsyslog.
I see. Using the default syslog-ng that ships with 12.04 (3.3.4+patches, as far as I remember) or 3.3.6+? I'd love to be able to fix the spinning issue, and perhaps I can do that faster than switching to rsyslog would be >;) -- |8]
Gergely Nagy <algernon <at> balabit.hu> writes:
I see. Using the default syslog-ng that ships with 12.04 (3.3.4+patches, as far as I remember) or 3.3.6+? I'd love to be able to fix the spinning issue, and perhaps I can do that faster than switching to rsyslog would be >;)
The package version is exactly "3.3.4.dfsg-2ubuntu1". There currently is no 3.3.6 in the repos, not even in unstable. If you think this can be fixed, I'd love to help, but it's difficult to reproduce.
Andreas Heinlein <aheinlein@gmx.com> writes:
Gergely Nagy <algernon <at> balabit.hu> writes:
I see. Using the default syslog-ng that ships with 12.04 (3.3.4+patches, as far as I remember) or 3.3.6+? I'd love to be able to fix the spinning issue, and perhaps I can do that faster than switching to rsyslog would be >;)
The package version is exactly "3.3.4.dfsg-2ubuntu1". There currently is no 3.3.6 in the repos, not even in unstable. If you think this can be fixed, I'd love to help, but it's difficult to reproduce.
I have packages built for pretty much every Ubuntu release starting from Lucid at: http://asylum.madhouse-project.org/projects/debian/ That repo has 3.3.7, Ubuntu 12.04 i386 & amd64 included. And yes, I believe that this can be fixed - if it is reproduced, and you can strace it for a little while at that point, that would be tremendous help to begin with (+ a config, so I can attempt reproducing too). -- |8]
Gergely Nagy <algernon <at> balabit.hu> writes:
I have packages built for pretty much every Ubuntu release starting from Lucid at: http://asylum.madhouse-project.org/projects/debian/
That repo has 3.3.7, Ubuntu 12.04 i386 & amd64 included.
Thanks a lot. I'll give it a try; I have installed it now on 3 clients. I'll report back when I find something.
----- Original message -----
Andreas Heinlein <aheinlein@gmx.com> writes:
we have a centralised log server running syslog-ng 3.1 OSE on Debian 6.0. On the client side, we were using syslog-ng but now I'd like to use rsyslog instead (for several reasons).
Independently of the issue below, I'd love to hear the reasons (either on-list, or in private).
Transport should be TLS-encrypted TCP. I have set up a connection between the two, but apparently syslog-ng fails to parse the log messages sent by rsyslog. Every log line goes like this:
Nov 6 11:15:31 admin2-desktop syslog-ng[1578]: Error processing log message: <13>Nov 6 11:15:31 admin2-desktop ah: Test4
Does anyone have an idea what to configure with either rsyslog or syslog-ng so the two understand each other?
Relevant server side config: source s_all { syslog(ip(172.16.x.x) port(6514) max_connections(50) tls( ^^^^^^
This is the issue. You're telling syslog-ng to expect the new syslog protocol, but later in the rsyslog.conf, you don't seem to be telling it to send that version, so it will use the legacy BSD format instead.
You have two options: either use tcp() on the syslog-ng side, or ask rsyslog to forward messages according to the new syslog protocol (however it may call it, it's RFC5424 by the way, while RFC3164 is the legacy BSD format).
I have updated the syslog() driver to automatically detect the rfc3164 format. but this happened in 3.3 or 3.4, can't remember which.
participants (3)
-
Andreas Heinlein
-
Balazs Scheidler
-
Gergely Nagy