[syslog-ng] Extremely slow receive on TCP

[Balazs Scheidler]

On May 7, 2013 11:37 PM, "Xuri Nagarin" <secsubs at gmail.com> wrote:
>
> Hi,
>
> I have a syslog-ng server that does not seem to receive well over TCP.
> Instead of an expected ~150 Mbits/s, I get a trickle of few events/sec.
> The setup is: Server L is an ArcSight Logger that streams out events
> over TCP or UDP. Server X is a test Syslog-ng VM that is running 3.2.5
> (from EPEL). Server P is running Syslog-NG 3.2.5 built from EPEL srpms
> with TLS enabled.
>
> L -> X over UDP Port 514, I get ~100 Mbit/sec
> L -> X over TCP Port 514, I get ~150 Mbit/sec
>
> L-> P over UDP Port 514, I get ~100 Mbit/sec
> L-> P over TCP Port 514, I get almost nothing. Just a few tens of events
> make it through over a half hour period.
>

I have a guess but that should only happen with 3.3 or later and should at
least display a warning message at startup.

Stating with 3.3 the flow control window calculation has changed. Syslog-ng
divides the window for each potential connection upfront.

You have max-connection set to 100 and log-iw-size defaults to 1000, which
means that syslog-ng allocates 10 to each connection which is somewhat low
in threaded mode.

Now as I think of it If you are really using 3.2 another issue may be the
cause, is there another client that sends a lot of traffic parallel to
logger? One source may starve the other. That's why the window allocation
was changed.

In any case try to increase that and see if it helps.

>  From L, I relay the same data over and over so there isn't an issue
> with what's being streamed.
>
> Config on X and P is identical and very basic:
> -----------------------------------------------------
> options {
>         time_reopen (10);
>         long_hostnames (off);
>         use_dns (no);
>         use_fqdn (no);
>         create_dirs (yes);
>         keep_hostname (yes);
>          keep_timestamp(yes);
>         log_fifo_size (1000);
>          stats_freq(60);
>          flush_timeout(10000);
>          log_fetch_limit(100);
>          flush_lines(100);
>          log_iw_size(100);
> };
>
> source s_tcp {
>          tcp( port(514) max-connections(100) );
> };
>
> source s_udp {
>          udp( port(514));
> };
>
> destination d_tcp { file("/var/log/tcp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); };
> destination d_udp { file("/var/log/udp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); };
> -----------------------------------------------------
>
>
> L -> X over TCP, L opens one TCP connections and everything comes over it.
> L -> P over TCP, L keeps opening up connections with P but no data comes
> over those connections or very few events come over those connections.
>
> I am going to investigate if there are any network QoS rules in place
> between L and P but wondering if there might be something in Syslog-NG
> that might be broken. I ran syslog-ng in debug mode but saw no errors. I
> looked at the pcap captures between L and P and saw no errors.
>
> Any other clues/ideas for troubleshooting are welcome.
>
> TIA!
>
>
>
______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130508/52a20eea/attachment-0001.htm