Dear list, Apologies for the long winded post but I'd really appreciate your comments on: 1) A configuration that did not perform adequately 2) My understanding of the relevant tuneables Number of clients: 40 Log sources per client: 35. 34 of these are pure file source driver, system logs use the standard redhat config. Dest: single loghost over tcp/ip, loghost uses mysql driver Due to other pressures I made no attempt to tune parameters that influence reliability before the test and performance was commensurately poor (about 40% of entries discarded at the client, confirmed by packet capture on the loghost). Client global opts: * log_msg_size (24576) * log_fifo_size (1000) * log_fetch_limit (10) * flush_lines(0) Client logging options: * each source does a program-override for loghost filtering, no other processing * flags(flow-control) NOT SET Loghost global opts * log_msg_size(32768) * log_fifo_size(1000) Loghost logging opts: * flags(flow-control) - NOT SET * streams undergoes a rewrite pass a simple filter (on program name), rewrite, parse, then mysql dest - the loghost load has always been negligable.
From what I NOW understand I need to do something like this:
Client: log_iw_size >= SOURCES_PER_CLIENT * log_fetch_limit eg 35 * 10 = 350 log_fifo_size >= SOURCES_PER_CLIENT * log_fetch_limit eg 35 * 10 = 350 AND log_fifo_size >= SOURCES_PER_CLIENT * log_iw_size eg 35 * 350 = 12250 So it appears to me that setting log_fifo_size to > 12250 would be correct. Loghost Less idea about this, do I need: log_iw_size >= NUMBER_OF_CLIENTS * log_fetch_limit ( * SOURCES_PER_CLIENT ? ) eg 40 * 10 * 35 = 14000 And log_fifo_size >= log_iw_size ? Is flow control important between the network source and the mysql dests? Further information: * Some of the larger logfiles output ~ 4 lines/sec * Approx 4G aggregate logs generated over 14 hours by the 40 hosts * Reliability is more important than speed - these logs are not analysed in real time. That said, the client should have to spend hours completing the log transfer once its workload has been processed.. Again, apologies for the length of the post and many thanks in advance for any help. Ben Tisdall PhotoBox