Hi, Thanks for the great description on how to reproduce the problem, it was a great help indeed. This bug has been part of syslog-ng for a couple of years now. This patch fixes it in the 3.3 tree. Could probably be applied to earlier versions easily. I'll do that once I get there, but no promises, since my 2nd child was born just yesterday :). Others on the list could perhaps help. commit dfc09fa50688600187f6c3e25f65a5c7877e924c Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Mar 20 18:27:16 2011 +0100 fixed syslog() source framed message reception If the frame length indicator is only partially read and positioned at the end of the buffer, syslog-ng would assume that the connection is closed, as it tries to read into a zero-length buffer. Thanks for Andy for creating such a detailed description on how to reproduce it here: https://lists.balabit.hu/pipermail/syslog-ng/2011-March/016240.html With that description it was easy to fix this nasty bug, which has been sitting in syslog-ng for all eternity. :) Reported-By: Andy Ruch <adruch2002@yahoo.com> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> On Thu, 2011-03-17 at 09:17 -0700, Andy Ruch wrote:
Hello,
I was evaluating syslog-ng and have come across a strange problem. When connecting to syslog-ng using the 'syslog' source, syslog-ng will return a TCP Reset at random intervals. The interesting thing about this error is it only seems to appear when the message length varies between logs. After some experimenting, I was able to reproduce the error using loggen to send logs from a file. However, when loggen generates its own fixed length messages, syslog-ng will receive everything properly.
I'm concerned about the reliability of syslog-ng. I don’t want it to close the connection if it gets overloaded. I know that I can add some options to improve performance, but that seems like a patch and not a fix. I would think syslog-ng would utilize TCP flow control to prevent the sender from sending too fast.
Am I doing something wrong? Does this look like a bug or have I just reached the max performance of syslog-ng? Any help is appreciated.
Below I’ve included some information about the commands that I was running as well as my system setup.
Thanks, Andy
***** Details *****
- Two machine setup (one sender and one receiver) - Receiver is using an evaluation copy of Syslog-PE-4.0.1a (but I first saw this issue using open source version 3.2.2 ) - CentOS 5.5
***** Send Command *****
This is the console output that I receive from loggen. As you can see, the failure did not happen every time. In this particular case, it was the second execution that failed. I normally just run it back to back until it fails.
$ ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514 average rate = 16503.22 msg/sec, count=50000, time=3.297, (last) msg size=112, bandwidth=1627.75 kB/sec $ ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514 Send error Connection reset by peer5 msg/sec average rate = 15483.69 msg/sec, count=15435, time=0.9968, (last) msg size=114, bandwidth=1527.27 kB/sec
***** Send File *****
The following pattern was put into a file and repeated for a total of 50,000 lines. I tried to attach the complete file but it was too large for this mailing list.
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
***** Syslog-ng.conf *****
@version: 4.0
options { };
###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel"));
};
source s_net { syslog( ip(0.0.0.0) transport("tcp") port(6514) ); };
###### # destinations destination d_messages { file("/var/log/messages"); };
destination d_tmp_file { file("/tmp/log_messages"); };
###### # paths log { source(s_local); destination(d_messages); };
log { source(s_net); destination(d_tmp_file); flags(flow-control); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi