[syslog-ng] Syslog-ng returning a TCP Reset

Balazs Scheidler bazsi at balabit.hu
Sun Mar 20 18:29:35 CET 2011 

Hi,

Thanks for the great description on how to reproduce the problem, it was
a great help indeed. This bug has been part of syslog-ng for a couple of
years now.

This patch fixes it in the 3.3 tree. Could probably be applied to
earlier versions easily. I'll do that once I get there, but no promises,
since my 2nd child was born just yesterday :). Others on the list could
perhaps help.


commit dfc09fa50688600187f6c3e25f65a5c7877e924c
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Sun Mar 20 18:27:16 2011 +0100

    fixed syslog() source framed message reception
    
    If the frame length indicator is only partially read
    and positioned at the end of the buffer, syslog-ng would assume that the
    connection is closed, as it tries to read into a zero-length buffer.
    
    Thanks for Andy for creating such a detailed description on how to
    reproduce it here:
    
    https://lists.balabit.hu/pipermail/syslog-ng/2011-March/016240.html
    
    With that description it was easy to fix this nasty bug, which has been
    sitting in syslog-ng for all eternity. :)
    
    Reported-By: Andy Ruch <adruch2002 at yahoo.com>
    Signed-off-by: Balazs Scheidler <bazsi at balabit.hu>




On Thu, 2011-03-17 at 09:17 -0700, Andy Ruch wrote:
> Hello,
> 
> I was evaluating syslog-ng and have come across a strange problem. When 
> connecting to syslog-ng using the 'syslog' source, syslog-ng will return a TCP 
> Reset at random intervals. The interesting thing about this error is it only 
> seems to appear when the message length varies between logs. After some 
> experimenting, I was able to reproduce the error using loggen to send logs from 
> a file. However, when loggen generates its own fixed length messages, syslog-ng 
> will receive everything properly.
> 
> I'm concerned about the reliability of syslog-ng. I don’t want it to close the 
> connection if it gets overloaded. I know that I can add some options to improve 
> performance, but that seems like a patch and not a fix. I would think syslog-ng 
> would utilize TCP flow control to prevent the sender from sending too fast.
> 
> Am I doing something wrong? Does this look like a bug or have I just reached the 
> max performance of syslog-ng? Any help is appreciated.
> 
> Below I’ve included some information about the commands that I was running as 
> well as my system setup.
> 
> Thanks,
> Andy
> 
> 
> 
> ***** Details *****
> 
>  - Two machine setup (one sender and one receiver)
>  - Receiver is using an evaluation copy of Syslog-PE-4.0.1a (but I first saw 
> this issue using open source version 3.2.2 )
>  - CentOS 5.5
> 
> ***** Send Command *****
> 
> This is the console output that I receive from loggen. As you can see, the 
> failure did not happen every time. In this particular case, it was the second 
> execution that failed. I normally just run it back to back until it fails.
> 
> $  ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514
> average rate = 16503.22 msg/sec, count=50000, time=3.297, (last) msg size=112, 
> bandwidth=1627.75 kB/sec
> $ ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514
> Send error Connection reset by peer5 msg/sec
> average rate = 15483.69 msg/sec, count=15435, time=0.9968, (last) msg size=114, 
> bandwidth=1527.27 kB/sec
> 
> ***** Send File *****
> 
> The following pattern was put into a file and repeated for a total of 50,000 
> lines. I tried to attach the complete file but it was too large for this mailing 
> list.
> 
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> 
> 
> 
> ***** Syslog-ng.conf *****
> 
> @version: 4.0
> 
> options {
> };
> 
> ######
> # sources
> source s_local {
> # message generated by Syslog-NG
> internal();
> # standard Linux log source (this is the default place for the syslog()
> # function to send logs to)
> unix-stream("/dev/log");
> # messages from the kernel
> file("/proc/kmsg" program_override("kernel"));
> 
> };
> 
> source s_net {
>     syslog( ip(0.0.0.0) transport("tcp") port(6514) );
> };
> 
> ######
> # destinations
> destination d_messages { file("/var/log/messages"); };
> 
> destination d_tmp_file { file("/tmp/log_messages"); };
> 
> ######
> # paths
> log {
> source(s_local);
> destination(d_messages);
> };
> 
> log {
>     source(s_net);
>     destination(d_tmp_file);
>     flags(flow-control);
> };
> 
> 
> 
>       
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 

-- 
Bazsi

More information about the syslog-ng mailing list