[syslog-ng] Extremely slow receive on TCP

[Xuri Nagarin]

Hi,

I have a syslog-ng server that does not seem to receive well over TCP. 
Instead of an expected ~150 Mbits/s, I get a trickle of few events/sec. 
The setup is: Server L is an ArcSight Logger that streams out events 
over TCP or UDP. Server X is a test Syslog-ng VM that is running 3.2.5 
(from EPEL). Server P is running Syslog-NG 3.2.5 built from EPEL srpms 
with TLS enabled.

L -> X over UDP Port 514, I get ~100 Mbit/sec
L -> X over TCP Port 514, I get ~150 Mbit/sec

L-> P over UDP Port 514, I get ~100 Mbit/sec
L-> P over TCP Port 514, I get almost nothing. Just a few tens of events 
make it through over a half hour period.

 From L, I relay the same data over and over so there isn't an issue 
with what's being streamed.

Config on X and P is identical and very basic:
-----------------------------------------------------
options {
	time_reopen (10);
	long_hostnames (off);
	use_dns (no);
	use_fqdn (no);
	create_dirs (yes);
	keep_hostname (yes);
         keep_timestamp(yes);
	log_fifo_size (1000);
         stats_freq(60);
         flush_timeout(10000);
         log_fetch_limit(100);
         flush_lines(100);
         log_iw_size(100);
};

source s_tcp {
         tcp( port(514) max-connections(100) );
};

source s_udp {
         udp( port(514));
};

destination d_tcp { file("/var/log/tcp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); };
destination d_udp { file("/var/log/udp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); };
-----------------------------------------------------


L -> X over TCP, L opens one TCP connections and everything comes over it.
L -> P over TCP, L keeps opening up connections with P but no data comes 
over those connections or very few events come over those connections.

I am going to investigate if there are any network QoS rules in place 
between L and P but wondering if there might be something in Syslog-NG 
that might be broken. I ran syslog-ng in debug mode but saw no errors. I 
looked at the pcap captures between L and P and saw no errors.

Any other clues/ideas for troubleshooting are welcome.

TIA!