[syslog-ng] Configuration tuning for reliability

[Martin Holste]

You should not be having problems with your load.  We had a thread
earlier this year ("UDP packet loss with syslog-ng") in which Lars
identified similar performance issues on RHEL.  His problems were
solved by setting the net.core.rmem_default to 2MB using sysctl.  I
would try setting that and then checking your performance.

On Thu, Nov 11, 2010 at 7:53 AM, Ben Tisdall <ben.tisdall at photobox.com> wrote:
> Dear list,
>
> Apologies for the long winded post but I'd really appreciate your comments on:
>
> 1) A configuration that did not perform adequately
> 2) My understanding of the relevant tuneables
>
> Number of clients: 40
> Log sources per client: 35. 34 of these are pure file source driver,
> system logs use the standard redhat config.
> Dest: single loghost over tcp/ip, loghost uses mysql driver
>
> Due to other pressures I made no attempt to tune parameters that
> influence reliability before the test and performance was
> commensurately poor (about 40% of entries discarded at the client,
> confirmed by packet capture on the loghost).
>
> Client global opts:
>
> * log_msg_size (24576)
> * log_fifo_size (1000)
> * log_fetch_limit (10)
> * flush_lines(0)
>
> Client logging options:
>
> * each source does a program-override for loghost filtering, no other processing
> * flags(flow-control) NOT SET
>
> Loghost global opts
>
> * log_msg_size(32768)
> * log_fifo_size(1000)
>
> Loghost logging opts:
>
> * flags(flow-control) - NOT SET
> * streams undergoes a rewrite pass a simple filter (on program name),
> rewrite, parse, then mysql dest - the loghost load has always been
> negligable.
>
> From what I NOW understand I need to do something like this:
>
> Client:
>
> log_iw_size >= SOURCES_PER_CLIENT * log_fetch_limit
>
> eg 35 * 10 = 350
>
> log_fifo_size >= SOURCES_PER_CLIENT * log_fetch_limit
>
> eg 35 * 10 = 350
>
> AND
>
> log_fifo_size >= SOURCES_PER_CLIENT * log_iw_size
>
> eg 35 * 350 = 12250
>
> So it appears to me that setting log_fifo_size to > 12250 would be correct.
>
> Loghost
>
> Less idea about this, do I need:
>
> log_iw_size >= NUMBER_OF_CLIENTS * log_fetch_limit ( * SOURCES_PER_CLIENT ? )
>
> eg 40 * 10 * 35 = 14000
>
> And log_fifo_size >= log_iw_size ?
>
> Is flow control important between the network source and the mysql dests?
>
> Further information:
>
> * Some of the larger logfiles output ~ 4 lines/sec
> * Approx 4G aggregate logs generated over 14 hours by the 40 hosts
> * Reliability is more important than speed - these logs are not
> analysed in real time. That said, the client should have to spend
> hours completing the log transfer once its workload has been
> processed..
>
> Again, apologies for the length of the post and many thanks in advance
> for any help.
>
> Ben Tisdall
> PhotoBox
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>