-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng- bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Monday, December 29, 2008 6:01 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-NG, log4j, tabs, whitespace, indent
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng- bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Thursday, December 11, 2008 7:21 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-NG, log4j, tabs, whitespace, indent
On Wed, 2008-12-10 at 16:55 -0600, John Skopis wrote:
Bazsi,
Strace: recvfrom(5, "<147> at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)", 8192, 0, {sa_family=AF_INET, sin_port=htons(41277), sin_addr=inet_addr("1.2.3.4")}, [16]) = 75 write(14, "Dec 10 17:52:04 1.2.3.4 at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)\n",
= 97
recvfrom(5, "<147> at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)", 8192, 0, {sa_family=AF_INET, sin_port=htons(41277), sin_addr=inet_addr("1.2.3.4")}, [16]) = 75 write(14, "Dec 10 17:52:04 1.2.3.4 at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:141)\n",
On Thu, 2008-12-11 at 10:37 -0600, John Skopis wrote: 97) 97)
= 97
Let me know if you need tcpdump still (I started the cap but
would like to anonymize it and/or send it to you off-list).
Ah, this means that there's no tab at the beginning of the message, but rather it is a space. syslog-ng skips spaces if the syslog header is not complete.
Try enabling setHeader(true) on your SyslogAppender class. This will probably make log4j emit a complete syslog header, in which case syslog-ng will not drop the preceding spaces.
Please let me know if this works for you.
Well, I tried setting Header=true but it did nothing. Actually, I think that it *is* setting the header for the first line of output, however as a stack trace is a log message spanning multiple lines it only sets the header for the first line.
hmm is log4j using TCP or UDP to send the message? I assume it is TCP.
The log4j version we are using does not support tcp. There is a nlog4j project but I am not sure we would to, or even be able to switch. I am not sure if any of this has changed in JBoss 5 as we are still on 4.
In TCP mode NL characters are the record terminator characters, so as it seems log4j generates a multiline message with only one header and several embedded NL characters. syslog-ng interprets this as a series of log messages, the first one with a syslog header the rest without one.
Is this also true for UDP? Is the multi-line message sent as multiple syslog datagrams (1 line = 1 datagram) or is it a single datagram with embedded newlines? I will need to check a packet capture to confirm.
Therefore space removal kicks in, the result is what you see.
Skipping the white-space removal is a band-aid solution, the real problem is that log4j uses multiline messages which syslog-ng does not support.
Agreed. It works though, in this case at least. I still don't understand why exactly the whitespace needs to be trimmed at all but I trust it's for a good reason.
The syslog-ng 3.0 branch already supports multi-line messages if the underlying transport supports those: either UDP or the new syslog-protocol. Are these real alternatives for you? I guess UDP is out of question because it may easily lose messages. The new syslog- protocol is not yet supported by log4j, however I know someone who implemented initial support for that.
I have not used syslog-ng 3.0 yet and our version of JBoss is not using the log4j with tcp support. How easily might the syslog server lose messages? It will never receive more than ~10000pps which it should easily be able to handle. Besides at that point it’s a *BSD or Linux problem and not a syslog-ng problem...right? Of course *BSD would never have such a problem to begin with. ;]
Would that be ok for you to switch over to use syslog-ng 3.0 and syslog-protocol?
I will try syslog-ng 3.x with standard UDP syslog messages. If it can handle the incomplete syslog message with whitespace a little better and is marked stable we would certainly consider switching. Thanks, John