[syslog-ng] delay in forwarding logs

[Balazs Scheidler]

On Wed, 2012-02-29 at 15:25 -0600, Ivanov, Konstantin wrote:
> I have two servers running syslog-ng 2.0.9 on Dell R710's. I did some
> benchmarking with loggen [1] and was able to generate 41000 msg/sec
> locally. When I try to forward log messages from one server to the
> other I only get about 60 msg/sec. tcpdump shows that only that many
> are sent over the wire as well. I test this by writing to the log file
> from a script. Any idea what can be the problem? Here are both configs
> from the sending and receiving servers:
> 
> 
> ----------- Sending syslog-ng server --------------
> 
> options {
> 
>     log_fifo_size(120000);
>     time_reopen (1);
>     time_reap(1);
>     long_hostnames (off);
>     use_dns (no);
>     use_fqdn (no);
>     create_dirs (yes);
>     group(adm);
>     perm(0640);
>     dir_perm(0755);
>     keep_hostname (yes);
>     log_fetch_limit(60000);
> 
> };
> 
> source s_zxtm {
>         file("/var/log/zxtm/access_log",follow_freq(1),
> flags(no-parse));
> };
> 
> destination d_logrelay { tcp(somehost.com port(514)); };
> 
> log {
>         source(s_zxtm); 
>         destination(d_logrelay);
>         flags(flow-control);
> };
> 
> --------------- Receiving syslog-ng server ------------------
> 
> options {
> 
>         log_fifo_size(3000000);
>         time_reopen (1);
>         time_reap(1);
>         long_hostnames (off);
>         use_dns (no);
>         use_fqdn (no);
>         create_dirs (yes);
>         group(adm);
>         perm(0640);
>         dir_perm(0755);
>         keep_hostname (yes);
>         stats_freq(600);
>         sync(0);
> 
> };
> 
> source s_zxtm {
>         tcp(max-connections(500), port(514));
> };
> 
> destination d_zxtm { file("/var/log/zxtm/incoming/$YEAR$MONTH$DAY
> $HOUR-access_log.aggregated" owner("glassfish") perm(0644)
> template("$MESSAGE\n") template_escape(no)); };
> 
> log {
>         source(s_zxtm);
>         destination(d_zxtm);
>         flags(flow-control);
> }; 
> 
> --------------
> 
> 
> 
> [1]. loggen -iS -r 10000000 -s 200 -I 60 localhost 514 average rate =
> 41053.50 msg/sec, count=410535

Hm... I'm not really sure what  the problem might be, 2.0.x is ancient
by all means. It might be related to flush_timeout() though, try to set
that to 0, or if at all possible try to upgrade to a newer version.

Hmm.. now for a second idea, I think you are bitten by the fact, that
you're checking the access log once every second (follow-freq(1)), and
in every iteration, syslog-ng only pulls in 8k of data (log-msg-size()
default), which is probably about 60 messages.

Also, file position tracking wasn't implemented in 2.0.x, so it wasn't
really good for tailing files like you do. This means that whenever you
restart syslog-ng, the file will be resent again.

-- 
Bazsi