[syslog-ng] [Bug 217] Poor single thraded performance reading from a pip

[bugzilla at bugzilla.balabit.com]

https://bugzilla.balabit.com/show_bug.cgi?id=217





--- Comment #2 from Evan Rempel <erempel at uvic.ca>  2013-01-22 07:00:04 ---
The best I can tell is that there is something wrong with the file destination in 3.4

Using a file destination seems to block everything. I can't get stats via
/usr/local/sbin/syslog-ng-ctl stats --control=/var/local/syslog-ng.ctl
because it just hangs.

It seems like it does not honour the flush_lines option, or if it does, it fsyncs every
line or something. The wait I/O goes to 40% on what is otherwise an idle box.
Even though I use a large (2,000,000) log_fifo_size

destination d_archive {
        file("/var/syslog/syslog.test.log" owner("root") group("syslogs") perm(0640) log_fifo_size(2000000) template(t_standardfile));
};

And have

source unix_network_tcp {
        tcp(localip(catamount.comp.uvic.ca) port(514) max_connections(10) log_fetch_limit(500) log_iw_size(1000000) );
        };

The destination will not queue up messages more than about 1000.
After 3 minutes of elapsed time, the first stats came back with
dst.file;d_archive#0;/var/syslog/syslog.test.log;a;dropped;0
dst.file;d_archive#0;/var/syslog/syslog.test.log;a;processed;6000
dst.file;d_archive#0;/var/syslog/syslog.test.log;a;stored;500


If I use a destination of
destination d_cat {
        program("/bin/cat >> /var/syslog/syslog.test.log" log_fifo_size(2000000) template(t_standardfile));
};

Then the messages are processed quickly and the CPU hits 100% for the syslog-ng process.
I also see the queued count reach 5,000 at times with this destination.


-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.