[syslog-ng] syslog-ng performance tuning

Fri Feb 19 09:27:34 CET 2010

Hi,

For those who are interested in, I solved my issue.

The problem was I had too many filter rules.
Using macros, I reduce about 600 rules to 3.

Now I get my syslog server working and no more lost messages.

Rémi

Rémi BUISSON wrote:
> I compiled version 2.1.14 but nothing has changed.
>
> I removed all my configuration and put configuration mentionned on 
> this blog: 
> http://bazsi.blogs.balabit.com/2007/12/syslog-ng-fun-with-performance.html
>
> syslog-ng-server:~# loggen -s 150 -r 100000 -S 127.0.0.1 2000
> average rate = 65539.50 msg/sec, count=655395
>
> syslog-ng-client:~# loggen -r 100000 -s 150 -i -S xxx.xxx.xxx.xxx 2000
> average rate = 22832.30 msg/sec, count=228323
>
> I wone 2 000 msg/sec upgrading my kernel to 2.6.26.
>
> Is there any TCP sysctl flag I can enable to make TCP connection to 
> syslog server better that you have in mind ?
>
>
> Rémi BUISSON wrote:
>> Siem,
>>
>> Thanks for trying helping me.
>>
>> My ulimit value was unlimited.
>> All my processes write <log$pid>m characters</log> so each process 
>> have its own n unique lines.
>>
>> I added a destination for my local5 which is the file /root/test.log.
>>
>> I tried: ./test_syslog.pl -p 5 -n 100 -m 1000
>>
>> on log client:
>> # wc -l /root/test.log
>> 500 test.log
>>
>> on log server:
>> # wc -l test.log
>> 0 test.log
>>
>> Then:
>>  ./test_syslog.pl -p 1000 -n 1000 -m 1000
>>
>> on log client:
>> # wc -l /root/test.log
>> 756688 test.log
>>
>> on log server:
>> # wc -l test.log
>> 9042 test.log
>>
>> The client outputs:
>> ...
>> Finished 9857!
>> ...
>> Finished 10904!
>> ...
>>
>> So randomly near the firsts and lasts processes spawned:
>>
>> client# grep 10904 test.log | wc -l
>> 0
>> client# grep 9857 test.log | wc -l
>> 1000
>>
>> server# grep 9857 test.log | wc -l
>> 4
>>
>> Sample of log:
>> Feb 15 10:01:05 xxxx logger: 
>> <log9857>000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 
>> 0000000000000000000000000</log>
>>
>> So, clearly the log server do not receive all logs but the client do 
>> not seem to be able to process a large amount of logging message.
>>
>> Each test result number is nearly the same. It's good to see there is 
>> no random in my tests ;-)
>>
>> Do you see the thing which make it not working ?
>>
>> Siem Korteweg wrote:
>>> Remi,
>>>
>>> just to make sure. Do your ulimit settings allow you to spawn the p (1000)
>>> processes in paralel?
>>>
>>> Considering your test. Did each instance of the test program write it's own
>>> unique lines and can you see whether some processes did not make it to syslog
>>> or that all processes produced partial logging?
>>>
>>> regards,
>>>
>>> Siem Korteweg
>>>
>>> -----Oorspronkelijk bericht-----
>>> Van: syslog-ng-bounces at lists.balabit.hu namens Rémi BUISSON
>>> Verzonden: vr 12-2-2010 17:51
>>> Aan: syslog-ng at lists.balabit.hu
>>> Onderwerp: [syslog-ng] syslog-ng performance tuning
>>>  
>>> Hi everybody,
>>>
>>> I'have an issue with syslog-ng configuration.
>>> I would like to centralize my logs on one server.
>>>
>>> I've a lot of logs to send. I don't know how many but I can estimate it 
>>> to 500GB per day from decades of servers.
>>> But, it writes only 25 GB per day.
>>> For some reasons I work on a debian etchnhalf environnement.
>>> So, I'm working with syslog-ng 2.0.0.
>>>
>>> I wrote a perl program which spawn p "logger -p local5.info" processes
>>> and send n lines of m characters.
>>>
>>> I'have tested with:
>>> p: 1 000
>>> n: 1 000
>>> m: 1 000
>>>
>>> Instead of having 1 000 000 lines in my logs I have nearly 10 000 lines !
>>> But my test was not revelant because normal logs where not stopped. So, 
>>> maybe normal.
>>>
>>>   
>>> ------------------------------------------------------------------------
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>   
>>
>> -- 
>> Rémi BUISSON - IT Engineer F-Secure Storage & Digital Content 7, rue 
>> Raymond Manaud
>> 33524 BORDEAUX Bruges Cedex
>> FRANCE 
>> http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png 
>>
>> ------------------------------------------------------------------------
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>   
>
> -- 
> Rémi BUISSON - IT Engineer F-Secure Storage & Digital Content 7, rue 
> Raymond Manaud
> 33524 BORDEAUX Bruges Cedex
> FRANCE 
> http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png 
>
> ------------------------------------------------------------------------
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>   

-- 
Rémi BUISSON - IT Engineer F-Secure Storage & Digital Content 7, rue 
Raymond Manaud
33524 BORDEAUX Bruges Cedex
FRANCE 
http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/45f9fc0e/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 3477 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/45f9fc0e/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 3477 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/45f9fc0e/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: f-secure.png
Type: image/png
Size: 3477 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/45f9fc0e/attachment-0005.png 