[syslog-ng] Performance issues

[farouk.s.khawaja at jpmorgan.com]

I'm finding that my Syslog-NG 2.0.9 OSE is not able to send more than 19 
msgs/sec.   I've tried setups with/without flow control, but keep coming 
up with similar performance results.  I'm using the 'file' driver to read 
the data from application log files.  I supposed I could use a named pipe, 
but I rather not create a new tool to write data to the pipe and create 
other bottlnecks and buffering issues.

The application logs I want to follow writes at a pace of ~130msgs/sec 
during peak usage with average message size of ~410bytes.  My destination 
log location is on NFS.  I've verified that it's not the network or NFS 
that's causing the issues.  I whipped up a simple perl script that made a 
socket connection from the source host to the destination host where 
syslog-ng was listening.  I was able to pump 48320 messages to that remote 
syslog port that was configured to write the data to a NFS partition. 
That's ~3200 msgs/sec.  So I also know that the destination syslog-ng is 
working fine.  My guess is that the problem lies with the file driver.

I'd like to get syslog-ng to be the tool I use to consolidate logs across 
N number of hosts instead of writing some bootleg code of my own to which 
I will forever be a slave to maintain.  Any assistance you can provide 
would be appreciated.

Source syslog-ng.conf

##########################################################
# G L O B A L  O P T I O N S 
##########################################################
options {
        use_dns (no);
        use_fqdn (no);
        create_dirs (yes);
        log_fifo_size(2100);
        time_sleep (10);
};

##########################################################
# Template
##########################################################
template t_clog {
        template("$MSG\n");
        template_escape(no);
};

##########################################################
# Sources
##########################################################
#
## default source for logging syslog messages
source s_internal {
        internal();
};

#
## 
source s_ldsol01_file {
        file("/tmp/ldsol01-clog.log"
             follow_freq(10)
             flags(no-parse)
             log_fetch_limit(1000)
             log_iw_size(2000)
        );
};


##########################################################
# Destinations
##########################################################
#
## default destination for logging syslog messages
destination d_syslog_file {
        file("/home/csh_apps/twataprd/Logs/SYSLOG-NG/syslog-ng.log");
};

#
## eqny-cshweb2
destination d_ldsol01_tcp {
        tcp( "10.3.22.43" port(20534) );
};
destination d_ldsol01rej_tcp {
        tcp("10.3.22.43" port(20535));
};

##########################################################
# logs
##########################################################
log { source( s_internal ); destination ( d_syslog_file ); };
log { source( s_ldsol01_file ); destination( d_ldsol01_tcp ); 
flags(flow-control); };






Generally, this communication is for informational purposes only
and it is not intended as an offer or solicitation for the purchase
or sale of any financial instrument or as an official confirmation
of any transaction. In the event you are receiving the offering
materials attached below related to your interest in hedge funds or
private equity, this communication may be intended as an offer or
solicitation for the purchase or sale of such fund(s).  All market
prices, data and other information are not warranted as to
completeness or accuracy and are subject to change without notice.
Any comments or statements made herein do not necessarily reflect
those of JPMorgan Chase & Co., its subsidiaries and affiliates.

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by JPMorgan Chase &
Co., its subsidiaries and affiliates, as applicable, for any loss
or damage arising in any way from its use. If you received this
transmission in error, please immediately contact the sender and
destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.
Please refer to http://www.jpmorgan.com/pages/disclosures for
disclosures relating to UK legal entities.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20081218/cb10b926/attachment.htm