hi, Attached you'll find a patch against syslog-ng 1.6.0rc4 which addresses some issues wrt log parsing. Please test if the patch fixes your problems. It might be part of a future release if it proves to interoperate with other platforms. People with not to frequent platforms (HP-UX comes to mind), please also test this patch as it might break there. Thanks. And finally the patch: Index: src/sources.c =================================================================== RCS file: /var/cvs/syslog-ng/syslog-ng/src/sources.c,v retrieving revision 1.37 diff -u -r1.37 sources.c --- src/sources.c 8 Jan 2003 09:31:37 -0000 1.37 +++ src/sources.c 3 Oct 2003 14:22:32 -0000 @@ -107,23 +107,32 @@ } closure->pos += n; + /* UDP packets are complete log lines on their own */ + if (closure->dgram) { + do_handle_line(closure, closure->pos, closure->buffer, salen ? (abstract_addr *) &sabuf : NULL, salen); + closure->pos = 0; + return ST_OK | ST_GOON; + } + eol = memchr(closure->buffer, '\0', closure->pos); - if (eol == NULL) - eol = memchr(closure->buffer, '\n', closure->pos); if (closure->pad_size && eol) { + + /* HP-UX sends chunks of pad_size in size, each containing a + * separate message, terminated by NUL */ do_handle_line(closure, eol - closure->buffer, closure->buffer, salen ? (abstract_addr *) &sabuf : NULL, salen); closure->pos = 0; return ST_OK | ST_GOON; } - if (!eol && (closure->dgram || closure->pos == closure->max_log_line)) { - /* we don't have a terminating nl nor \0, and our buffer is - full or we are a datagram receiver, when the message is in - its own packet. - */ + if (eol == NULL) + eol = memchr(closure->buffer, '\n', closure->pos); + if (!eol && closure->pos == closure->max_log_line) { + /* we don't have a terminating nl nor \0, and our buffer is full */ do_handle_line(closure, closure->pos, closure->buffer, salen ? (abstract_addr *) &sabuf : NULL, salen); closure->pos = 0; return ST_OK | ST_GOON; } + + /* iterate through lines */ start = closure->buffer; while (eol) { /* eol points at the newline character. end points at the -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
Hi, Quote Balazs Scheidler <bazsi@balabit.hu>: | | Attached you'll find a patch against syslog-ng 1.6.0rc4 which addresses | some issues wrt log parsing. Please test if the patch fixes your problems. I've juste tested the patch, and quickly noticed that received messages are written to the log files with no newline appended, they all end up on the same line. I also tested without the patch applied: the problem does not occur. All this is with UDP messages. I will happily make more tests if you send me a fixed patch. Regards, -Thomas PS: I had to use "patch -l" to use your patch, some tabs were probably lost during a copy-paste.
On Mon, Oct 06, 2003 at 01:38:16PM +0200, Thomas Morin wrote:
Hi,
Quote Balazs Scheidler <bazsi@balabit.hu>: | | Attached you'll find a patch against syslog-ng 1.6.0rc4 which addresses | some issues wrt log parsing. Please test if the patch fixes your problems.
I've juste tested the patch, and quickly noticed that received messages are written to the log files with no newline appended, they all end up on the same line.
I also tested without the patch applied: the problem does not occur.
I've tested the following config: source src { udp(port(2000)); internal(); }; destination dst { file("logfile"); }; log { source(src); destination(dst); }; And I can't seem to reproduce the problem: bzorp:src$ ./syslog-ng -dvf syslog-ng.conf binding fd 3, inetaddr: 0.0.0.0, port: 2000 io.c: Preparing fd 3 for reading syslog-ng version 1.6.0rc4 starting io.c: Preparing fd 4 for writing ^C bzorp:src$ cat logfile Oct 6 15:27:27 localhost/localhost message1 Oct 6 15:27:28 localhost/localhost message2 Oct 6 15:27:29 localhost/localhost message3 -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
Quote Balazs Scheidler <bazsi@balabit.hu>: | | I've tested the following config: | | source src { udp(port(2000)); internal(); }; | | destination dst { file("logfile"); }; | | log { source(src); destination(dst); }; | | And I can't seem to reproduce the problem I probably should have said that I'm not sending syslog messages with a syslog "forwarder" daemon nor with a standard C API, and that the messages I sent are _not_ newline terminated (they are NULL terminated though). I believe that messages send by standard syslog API (or syslog forwarder servers) are newline terminated, so I guess this is were lies the difference. If needed I can send you the perl proggie that I use to send those logs. Regards, -Thomas -- == Thomas Morin == Ingénieur Consultant Atlantide - www.ago.fr - thomas.morin@ago.fr == PGP Id:8CEA233D Key FP:503BF6CFD3AE8719377B832A02FB94E08CEA233D --
Hi, I've finally gotten to allocate some time releasing the final syslog-ng 1.6.0, and while collecting the bits I've found our discussion back in October 2003, when I posted a patch which should have fixed a log4j interoperability problem. You said messages were not newline terminated, I replied I couldn't reproduce the problem and you answered the difference might be that you are sending messages with a custom perl script which does not end lines with newlines whereas my testcases did. I now retested the code, this time without newlines as the following strace snippet shows: $ echo -n 'valami uzenet' | strace nc -u localhost 2000 .
Hi Balazs, Balazs Scheidler wrote:
I've finally gotten to allocate some time releasing the final syslog-ng 1.6.0, and while collecting the bits I've found our discussion back in October 2003, when I posted a patch which should have fixed a log4j interoperability problem.
You said messages were not newline terminated, I replied I couldn't reproduce the problem and you answered the difference might be that you are sending messages with a custom perl script which does not end lines with newlines whereas my testcases did.
I now retested the code, this time without newlines as the following strace snippet shows:
Well I hope the the bug is now fixed. I'm not in position to test this anymore : I work for some IT consulting company, and I'm not at the place I was working back in october. At the time our fix had been to modify the log4j code to newline terminate the messages, which was fine as a first step. Only messages bigger than syslog-ng message buffers were not properly parsed (newline was eaten because buffer too small); so we added a limit to the message size in log4j, which closed the problem for us. I'll notify the guys working there anyway, telling them a corrected version of syslog-ng had been released. Thanks a lot for your work, I already told it, but I'll repeat it anyawys : syslog-ng is a great piece of software ! :) Happy new year, -Thomas
participants (3)
-
Balazs Scheidler
-
Thomas Morin
-
Thomas Morin