[syslog-ng] Syslog-NG, log4j, tabs, whitespace, indent

[John Skopis]

> -----Original Message-----
> From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-
> bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
> Sent: Monday, December 29, 2008 6:01 AM
> To: Syslog-ng users' and developers' mailing list
> Subject: Re: [syslog-ng] Syslog-NG, log4j, tabs, whitespace, indent
> 
> On Thu, 2008-12-11 at 10:37 -0600, John Skopis wrote:
> > > -----Original Message-----
> > > From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-
> > > bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
> > > Sent: Thursday, December 11, 2008 7:21 AM
> > > To: Syslog-ng users' and developers' mailing list
> > > Subject: Re: [syslog-ng] Syslog-NG, log4j, tabs, whitespace, indent
> > >
> > > On Wed, 2008-12-10 at 16:55 -0600, John Skopis wrote:
> > > > Bazsi,
> > > >
> > > > Strace:
> > > > recvfrom(5, "<147>    at
> > > org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)",
> 8192,
> > > 0, {sa_family=AF_INET, sin_port=htons(41277),
> > > sin_addr=inet_addr("1.2.3.4")}, [16]) = 75
> > > > write(14, "Dec 10 17:52:04 1.2.3.4 at
> > > org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)\n",
> 97)
> > > = 97
> > > > recvfrom(5, "<147>    at
> > > org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)",
> 8192,
> > > 0, {sa_family=AF_INET, sin_port=htons(41277),
> > > sin_addr=inet_addr("1.2.3.4")}, [16]) = 75
> > > > write(14, "Dec 10 17:52:04 1.2.3.4 at
> > > org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)\n",
> 97)
> > > = 97
> > > >
> > > > Let me know if you need tcpdump still (I started the cap but
> would
> > > like to anonymize it and/or send it to you off-list).
> > > >
> > >
> > > Ah, this means that there's no tab at the beginning of the message,
> but
> > > rather it is a space. syslog-ng skips spaces if the syslog header
> is
> > > not
> > > complete.
> > >
> > > Try enabling setHeader(true) on your SyslogAppender class. This
> will
> > > probably make log4j emit a complete syslog header, in which case
> > > syslog-ng will not drop the preceding spaces.
> > >
> > > Please let me know if this works for you.
> >
> > Well, I tried setting Header=true but it did nothing. Actually, I
> think that it *is* setting the
> > header for the first line of output, however as a stack trace is a
> log message spanning multiple
> > lines it only sets the header for the first line.
> 
> hmm is log4j using TCP or UDP to send the message? I assume it is TCP.
> 

The log4j version we are using does not support tcp. There is a nlog4j project but I am not sure we would to, or even be able to switch. I am not sure if any of this has changed in JBoss 5 as we are still on 4.

> In TCP mode NL characters are the record terminator characters, so as
> it
> seems log4j generates a multiline message with only one header and
> several embedded NL characters. syslog-ng interprets this as a series
> of
> log messages, the first one with a syslog header the rest without one.
> 

Is this also true for UDP? Is the multi-line message sent as multiple syslog datagrams (1 line = 1 datagram) or is it a single datagram with embedded newlines? I will need to check a packet capture to confirm.

> Therefore space removal kicks in, the result is what you see.
> 
> Skipping the white-space removal is a band-aid solution, the real
> problem is that log4j uses multiline messages which syslog-ng does not
> support.

Agreed. It works though, in this case at least. I still don't understand why exactly the whitespace needs to be trimmed at all but I trust it's for a good reason.

> 
> The syslog-ng 3.0 branch already supports multi-line messages if the
> underlying transport supports those: either UDP or the new
> syslog-protocol. Are these real alternatives for you? I guess UDP is
> out
> of question because it may easily lose messages. The new syslog-
> protocol
> is not yet supported by log4j, however I know someone who implemented
> initial support for that.
> 

I have not used syslog-ng 3.0 yet and our version of JBoss is not using the log4j with tcp support.

How easily might the syslog server lose messages? It will never receive more than ~10000pps which it should easily be able to handle. Besides at that point it’s a *BSD or Linux problem and not a syslog-ng problem...right? Of course *BSD would never have such a problem to begin with. ;]

> Would that be ok for you to switch over to use syslog-ng 3.0 and
> syslog-protocol?
> 

I will try syslog-ng 3.x with standard UDP syslog messages. If it can handle the incomplete syslog message with whitespace a little better and is marked stable we would certainly consider switching.

Thanks,
John