[syslog-ng] Syslog-ng Drops Logs

[Sandor Geller]

Hi,

On 11/05/2014 04:23 PM, Yalin Aksoy wrote:
> We have lots of traffic going through syslog-ng in our system (5000 logs
> per second), so some logs are dropped because of 'log_fifo_size()'.

I think there is some misunderstanding here. log_fifo_size() controls 
how many logs could get buffered if a destination can't process the logs 
fast enough - either because the destination has a bottleneck (database, 
TCP-based network destinations could be such for example) or the 
destination driver doesn't get called frequently enough to flush the 
logs when you've got many destinations.

> I've look around the web and found flush-lines and flush-timeout
> methods, but also it failed in every configuration.
> Related parts of my syslog-ng conf looks like that at the moment.
> "
> log_fifo_size(4096);

As mentioned above this option controls how big the overflow buffer 
would be. This option could get used globally and per destination as 
well. Multiply it with the maximal message size (8kB by default) and the 
number of destinations to get the theoretical upper limit how much 
memory syslog-ng could use for internal message buffering. Increasing 
this setting could handle small bursts but won't help when the average 
incoming message rate exceeds what the destinations could handle.

> flush_lines(100);

This setting controls how many logs should a destination driver flush in 
a single iteration. Increasing this number could increase performance 
but with slow destinations it could also lead to delays.

> flush_timeout(1000);

This is a currently deprecated option and has effect only when there is 
less than flush_lines() amount of logs in the buffer so increasing it 
won't help with busy destinations.

> If I increase fifo size to ~ 16000 ,syslog-ng consumes too much memory
> for my system to operate.

About 128MB per destination, although not small but pretty much 
affordable for busy log relays, if your incoming log rate is 5k/sec then 
this buffer lasts only to 3.2 seconds. Not much at all...

As syslog-ng shows which destinations drop messages you can try 
increasing the buffer size to the problematic destination only.

> Is there any other way to stop that leak and what is the best practice
> to use?

You should check flow control-related parts of the syslog-ng OSE admin 
guide to see your options. If you're using non-flow-controllable sources 
or a destination is really slow then your system simply might not be 
powerful enough to process the load.

BTW which syslog-ng version are you using? Recent, multi-threaded ones 
should handle certain loads much-much better than older versions. Also 
showing your config might help as there may be parts which could get 
updated to give better performance. Without understanding your usecase 
in detail we could only guess.

Regards,

Sandor