[syslog-ng] Syslog-ng returning a TCP Reset

[Andy Ruch]

Hello,

I was evaluating syslog-ng and have come across a strange problem. When 
connecting to syslog-ng using the 'syslog' source, syslog-ng will return a TCP 
Reset at random intervals. The interesting thing about this error is it only 
seems to appear when the message length varies between logs. After some 
experimenting, I was able to reproduce the error using loggen to send logs from 
a file. However, when loggen generates its own fixed length messages, syslog-ng 
will receive everything properly.

I'm concerned about the reliability of syslog-ng. I don’t want it to close the 
connection if it gets overloaded. I know that I can add some options to improve 
performance, but that seems like a patch and not a fix. I would think syslog-ng 
would utilize TCP flow control to prevent the sender from sending too fast.

Am I doing something wrong? Does this look like a bug or have I just reached the 
max performance of syslog-ng? Any help is appreciated.

Below I’ve included some information about the commands that I was running as 
well as my system setup.

Thanks,
Andy



***** Details *****

 - Two machine setup (one sender and one receiver)
 - Receiver is using an evaluation copy of Syslog-PE-4.0.1a (but I first saw 
this issue using open source version 3.2.2 )
 - CentOS 5.5

***** Send Command *****

This is the console output that I receive from loggen. As you can see, the 
failure did not happen every time. In this particular case, it was the second 
execution that failed. I normally just run it back to back until it fails.

$  ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514
average rate = 16503.22 msg/sec, count=50000, time=3.297, (last) msg size=112, 
bandwidth=1627.75 kB/sec
$ ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514
Send error Connection reset by peer5 msg/sec
average rate = 15483.69 msg/sec, count=15435, time=0.9968, (last) msg size=114, 
bandwidth=1527.27 kB/sec

***** Send File *****

The following pattern was put into a file and repeated for a total of 50,000 
lines. I tried to attach the complete file but it was too large for this mailing 
list.

Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Mar 16 10:35:28 guinness syslog-tester[proc-1234]: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx



***** Syslog-ng.conf *****

@version: 4.0

options {
};

######
# sources
source s_local {
# message generated by Syslog-NG
internal();
# standard Linux log source (this is the default place for the syslog()
# function to send logs to)
unix-stream("/dev/log");
# messages from the kernel
file("/proc/kmsg" program_override("kernel"));

};

source s_net {
    syslog( ip(0.0.0.0) transport("tcp") port(6514) );
};

######
# destinations
destination d_messages { file("/var/log/messages"); };

destination d_tmp_file { file("/tmp/log_messages"); };

######
# paths
log {
source(s_local);
destination(d_messages);
};

log {
    source(s_net);
    destination(d_tmp_file);
    flags(flow-control);
};