Siem, Thanks for trying helping me. My ulimit value was unlimited. All my processes write <log$pid>m characters</log> so each process have its own n unique lines. I added a destination for my local5 which is the file /root/test.log. I tried: ./test_syslog.pl -p 5 -n 100 -m 1000 on log client: # wc -l /root/test.log 500 test.log on log server: # wc -l test.log 0 test.log Then: ./test_syslog.pl -p 1000 -n 1000 -m 1000 on log client: # wc -l /root/test.log 756688 test.log on log server: # wc -l test.log 9042 test.log The client outputs: ... Finished 9857! ... Finished 10904! ... So randomly near the firsts and lasts processes spawned: client# grep 10904 test.log | wc -l 0 client# grep 9857 test.log | wc -l 1000 server# grep 9857 test.log | wc -l 4 Sample of log: Feb 15 10:01:05 xxxx logger: <log9857>0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000</log> So, clearly the log server do not receive all logs but the client do not seem to be able to process a large amount of logging message. Each test result number is nearly the same. It's good to see there is no random in my tests ;-) Do you see the thing which make it not working ? Siem Korteweg wrote:
Remi,
just to make sure. Do your ulimit settings allow you to spawn the p (1000) processes in paralel?
Considering your test. Did each instance of the test program write it's own unique lines and can you see whether some processes did not make it to syslog or that all processes produced partial logging?
regards,
Siem Korteweg
-----Oorspronkelijk bericht----- Van: syslog-ng-bounces@lists.balabit.hu namens Rémi BUISSON Verzonden: vr 12-2-2010 17:51 Aan: syslog-ng@lists.balabit.hu Onderwerp: [syslog-ng] syslog-ng performance tuning
Hi everybody,
I'have an issue with syslog-ng configuration. I would like to centralize my logs on one server.
I've a lot of logs to send. I don't know how many but I can estimate it to 500GB per day from decades of servers. But, it writes only 25 GB per day. For some reasons I work on a debian etchnhalf environnement. So, I'm working with syslog-ng 2.0.0.
I wrote a perl program which spawn p "logger -p local5.info" processes and send n lines of m characters.
I'have tested with: p: 1 000 n: 1 000 m: 1 000
Instead of having 1 000 000 lines in my logs I have nearly 10 000 lines ! But my test was not revelant because normal logs where not stopped. So, maybe normal.
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Rémi BUISSON - IT Engineer F-Secure Storage & Digital Content 7, rue Raymond Manaud 33524 BORDEAUX Bruges Cedex FRANCE http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/...
