Hello, Thank you so much for detailed and rapid response. We could listen all the log with another destination. So probably my program is not fast enough. I'm going to update it to work multithreaded. I'll post the solution and cause of the problem afterwards. Regards, -Yalin On 05-11-2014 18:00, Sandor Geller wrote:
Hi,
On 11/05/2014 04:23 PM, Yalin Aksoy wrote:
We have lots of traffic going through syslog-ng in our system (5000 logs per second), so some logs are dropped because of 'log_fifo_size()'. I think there is some misunderstanding here. log_fifo_size() controls how many logs could get buffered if a destination can't process the logs fast enough - either because the destination has a bottleneck (database, TCP-based network destinations could be such for example) or the destination driver doesn't get called frequently enough to flush the logs when you've got many destinations.
I've look around the web and found flush-lines and flush-timeout methods, but also it failed in every configuration. Related parts of my syslog-ng conf looks like that at the moment. " log_fifo_size(4096); As mentioned above this option controls how big the overflow buffer would be. This option could get used globally and per destination as well. Multiply it with the maximal message size (8kB by default) and the number of destinations to get the theoretical upper limit how much memory syslog-ng could use for internal message buffering. Increasing this setting could handle small bursts but won't help when the average incoming message rate exceeds what the destinations could handle.
flush_lines(100); This setting controls how many logs should a destination driver flush in a single iteration. Increasing this number could increase performance but with slow destinations it could also lead to delays.
flush_timeout(1000); This is a currently deprecated option and has effect only when there is less than flush_lines() amount of logs in the buffer so increasing it won't help with busy destinations.
If I increase fifo size to ~ 16000 ,syslog-ng consumes too much memory for my system to operate. About 128MB per destination, although not small but pretty much affordable for busy log relays, if your incoming log rate is 5k/sec then this buffer lasts only to 3.2 seconds. Not much at all...
As syslog-ng shows which destinations drop messages you can try increasing the buffer size to the problematic destination only.
Is there any other way to stop that leak and what is the best practice to use? You should check flow control-related parts of the syslog-ng OSE admin guide to see your options. If you're using non-flow-controllable sources or a destination is really slow then your system simply might not be powerful enough to process the load.
BTW which syslog-ng version are you using? Recent, multi-threaded ones should handle certain loads much-much better than older versions. Also showing your config might help as there may be parts which could get updated to give better performance. Without understanding your usecase in detail we could only guess.
Regards,
Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq