[syslog-ng] syslog-ng performance tuning

Fri Feb 19 21:36:24 CET 2010

Do you think you can you share your modified config? thanks

On Fri, Feb 19, 2010 at 2:27 AM, Rémi BUISSON <rbuisson at steek.com> wrote:

>  Hi,
>
> For those who are interested in, I solved my issue.
>
> The problem was I had too many filter rules.
> Using macros, I reduce about 600 rules to 3.
>
> Now I get my syslog server working and no more lost messages.
>
> Rémi
>
>
> Rémi BUISSON wrote:
>
> I compiled version 2.1.14 but nothing has changed.
>
> I removed all my configuration and put configuration mentionned on this
> blog:
> http://bazsi.blogs.balabit.com/2007/12/syslog-ng-fun-with-performance.html
>
> syslog-ng-server:~# loggen -s 150 -r 100000 -S 127.0.0.1 2000
> average rate = 65539.50 msg/sec, count=655395
>
> syslog-ng-client:~# loggen -r 100000 -s 150 -i -S xxx.xxx.xxx.xxx 2000
> average rate = 22832.30 msg/sec, count=228323
>
> I wone 2 000 msg/sec upgrading my kernel to 2.6.26.
>
> Is there any TCP sysctl flag I can enable to make TCP connection to syslog
> server better that you have in mind ?
>
>
> Rémi BUISSON wrote:
>
> Siem,
>
> Thanks for trying helping me.
>
> My ulimit value was unlimited.
> All my processes write <log$pid>m characters</log> so each process have its
> own n unique lines.
>
> I added a destination for my local5 which is the file /root/test.log.
>
> I tried: ./test_syslog.pl -p 5 -n 100 -m 1000
>
> on log client:
> # wc -l /root/test.log
> 500 test.log
>
> on log server:
> # wc -l test.log
> 0 test.log
>
> Then:
>  ./test_syslog.pl -p 1000 -n 1000 -m 1000
>
> on log client:
> # wc -l /root/test.log
> 756688 test.log
>
> on log server:
> # wc -l test.log
> 9042 test.log
>
> The client outputs:
> ...
> Finished 9857!
> ...
> Finished 10904!
> ...
>
> So randomly near the firsts and lasts processes spawned:
>
> client# grep 10904 test.log | wc -l
> 0
> client# grep 9857 test.log | wc -l
> 1000
>
> server# grep 9857 test.log | wc -l
> 4
>
> Sample of log:
> Feb 15 10:01:05 xxxx logger:
> <log9857>000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> 0000000000000000000000000</log>
>
> So, clearly the log server do not receive all logs but the client do not
> seem to be able to process a large amount of logging message.
>
> Each test result number is nearly the same. It's good to see there is no
> random in my tests ;-)
>
> Do you see the thing which make it not working ?
>
> Siem Korteweg wrote:
>
> Remi,
>
> just to make sure. Do your ulimit settings allow you to spawn the p (1000)
> processes in paralel?
>
> Considering your test. Did each instance of the test program write it's own
> unique lines and can you see whether some processes did not make it to syslog
> or that all processes produced partial logging?
>
> regards,
>
> Siem Korteweg
>
> -----Oorspronkelijk bericht-----
> Van: syslog-ng-bounces at lists.balabit.hu namens Rémi BUISSON
> Verzonden: vr 12-2-2010 17:51
> Aan: syslog-ng at lists.balabit.hu
> Onderwerp: [syslog-ng] syslog-ng performance tuning
>
> Hi everybody,
>
> I'have an issue with syslog-ng configuration.
> I would like to centralize my logs on one server.
>
> I've a lot of logs to send. I don't know how many but I can estimate it
> to 500GB per day from decades of servers.
> But, it writes only 25 GB per day.
> For some reasons I work on a debian etchnhalf environnement.
> So, I'm working with syslog-ng 2.0.0.
>
> I wrote a perl program which spawn p "logger -p local5.info" processes
> and send n lines of m characters.
>
> I'have tested with:
> p: 1 000
> n: 1 000
> m: 1 000
>
> Instead of having 1 000 000 lines in my logs I have nearly 10 000 lines !
> But my test was not revelant because normal logs where not stopped. So,
> maybe normal.
>
>
>
> ------------------------------
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
> --
>  Rémi BUISSON - IT Engineer F-Secure Storage & Digital Content 7, rue
> Raymond Manaud
> 33524 BORDEAUX Bruges Cedex
> FRANCE [image:
> http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png]
>
> ------------------------------
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
> --
>  Rémi BUISSON - IT Engineer F-Secure Storage & Digital Content 7, rue
> Raymond Manaud
> 33524 BORDEAUX Bruges Cedex
> FRANCE [image:
> http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png]
>
> ------------------------------
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
> --
>  Rémi BUISSON - IT Engineer F-Secure Storage & Digital Content 7, rue
> Raymond Manaud
> 33524 BORDEAUX Bruges Cedex
> FRANCE [image:
> http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png]
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/885bbee8/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 3477 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/885bbee8/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 3477 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/885bbee8/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 3477 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100219/885bbee8/attachment-0005.png 