I'm finding that my Syslog-NG 2.0.9
OSE is not able to send more than 19 msgs/sec. I've tried setups
with/without flow control, but keep coming up with similar performance
results. I'm using the 'file' driver to read the data from application
log files. I supposed I could use a named pipe, but I rather not
create a new tool to write data to the pipe and create other bottlnecks
and buffering issues.
The application logs I want to follow
writes at a pace of ~130msgs/sec during peak usage with average message
size of ~410bytes. My destination log location is on NFS. I've
verified that it's not the network or NFS that's causing the issues. I
whipped up a simple perl script that made a socket connection from the
source host to the destination host where syslog-ng was listening. I
was able to pump 48320 messages to that remote syslog port that was configured
to write the data to a NFS partition. That's ~3200 msgs/sec. So
I also know that the destination syslog-ng is working fine. My guess
is that the problem lies with the file driver.
I'd like to get syslog-ng to be the
tool I use to consolidate logs across N number of hosts instead of writing
some bootleg code of my own to which I will forever be a slave to maintain.
Any assistance you can provide would be appreciated.
Source syslog-ng.conf
##########################################################
# G L O B A L O P T I O N S
##########################################################
options {
use_dns
(no);
use_fqdn
(no);
create_dirs
(yes);
log_fifo_size(2100);
time_sleep
(10);
};
##########################################################
# Template
##########################################################
template t_clog {
template("$MSG\n");
template_escape(no);
};
##########################################################
# Sources
##########################################################
#
## default source for logging syslog
messages
source s_internal {
internal();
};
#
##
source s_ldsol01_file {
file("/tmp/ldsol01-clog.log"
follow_freq(10)
flags(no-parse)
log_fetch_limit(1000)
log_iw_size(2000)
);
};
##########################################################
# Destinations
##########################################################
#
## default destination for logging syslog
messages
destination d_syslog_file {
file("/home/csh_apps/twataprd/Logs/SYSLOG-NG/syslog-ng.log");
};
#
## eqny-cshweb2
destination d_ldsol01_tcp {
tcp( "10.3.22.43"
port(20534) );
};
destination d_ldsol01rej_tcp {
tcp("10.3.22.43"
port(20535));
};
##########################################################
# logs
##########################################################
log { source( s_internal ); destination
( d_syslog_file ); };
log { source( s_ldsol01_file ); destination(
d_ldsol01_tcp ); flags(flow-control); };
Generally, this communication is for informational purposes only and it is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. In the event you are receiving the offering materials attached below related to your interest in hedge funds or private equity, this communication may be intended as an offer or solicitation for the purchase or sale of such fund(s). All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates.
This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you.
Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to UK legal entities.