Do you think you can you share your modified config? thanks<br><br><div class="gmail_quote">On Fri, Feb 19, 2010 at 2:27 AM, Rémi BUISSON <span dir="ltr"><<a href="mailto:rbuisson@steek.com">rbuisson@steek.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">
Hi,<br>
<br>
For those who are interested in, I solved my issue.<br>
<br>
The problem was I had too many filter rules.<br>
Using macros, I reduce about 600 rules to 3.<br>
<br>
Now I get my syslog server working and no more lost messages.<br><font color="#888888">
<br>
Rémi</font><div><div></div><div class="h5"><br>
<br>
Rémi BUISSON wrote:
<blockquote type="cite">
I compiled version 2.1.14 but nothing has changed.<br>
<br>
I removed all my configuration and put configuration mentionned on this
blog:
<a href="http://bazsi.blogs.balabit.com/2007/12/syslog-ng-fun-with-performance.html" target="_blank">http://bazsi.blogs.balabit.com/2007/12/syslog-ng-fun-with-performance.html</a><br>
<br>
syslog-ng-server:~# loggen -s 150 -r 100000 -S 127.0.0.1 2000<br>
average rate = 65539.50 msg/sec, count=655395<br>
<br>
syslog-ng-client:~# loggen -r 100000 -s 150 -i -S xxx.xxx.xxx.xxx 2000<br>
average rate = 22832.30 msg/sec, count=228323<br>
<br>
I wone 2 000 msg/sec upgrading my kernel to 2.6.26.<br>
<br>
Is there any TCP sysctl flag I can enable to make TCP connection to
syslog server better that you have in mind ?<br>
<br>
<br>
Rémi BUISSON wrote:
<blockquote type="cite">
Siem,<br>
<br>
Thanks for trying helping me.<br>
<br>
My ulimit value was unlimited.<br>
All my processes write <log$pid>m characters</log> so each
process have its own n unique lines.<br>
<br>
I added a destination for my local5 which is the file /root/test.log.<br>
<br>
I tried: ./<a href="http://test_syslog.pl" target="_blank">test_syslog.pl</a> -p 5 -n 100 -m 1000<br>
<br>
on log client:<br>
# wc -l /root/test.log<br>
500 test.log<br>
<br>
on log server:<br>
# wc -l test.log <br>
0 test.log<br>
<br>
Then:<br>
./<a href="http://test_syslog.pl" target="_blank">test_syslog.pl</a> -p 1000 -n 1000 -m 1000<br>
<br>
on log client:<br>
# wc -l /root/test.log<br>
756688 test.log<br>
<br>
on log server:<br>
# wc -l test.log <br>
9042 test.log<br>
<br>
The client outputs:<br>
...<br>
Finished 9857!<br>
...<br>
Finished 10904!<br>
...<br>
<br>
So randomly near the firsts and lasts processes spawned:<br>
<br>
client# grep 10904 test.log | wc -l<br>
0<br>
client# grep 9857 test.log | wc -l<br>
1000<br>
<br>
server# grep 9857 test.log | wc -l<br>
4<br>
<br>
Sample of log:<br>
Feb 15 10:01:05 xxxx logger:
<log9857>000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000</log><br>
<br>
So, clearly the log server do not receive all logs but the client do
not seem to be able to process a large amount of logging message.<br>
<br>
Each test result number is nearly the same. It's good to see there is
no random in my tests ;-)<br>
<br>
Do you see the thing which make it not working ?<br>
<br>
Siem Korteweg wrote:
<blockquote type="cite">
<pre>Remi,
just to make sure. Do your ulimit settings allow you to spawn the p (1000)
processes in paralel?
Considering your test. Did each instance of the test program write it's own
unique lines and can you see whether some processes did not make it to syslog
or that all processes produced partial logging?
regards,
Siem Korteweg
-----Oorspronkelijk bericht-----
Van: <a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a> namens Rémi BUISSON
Verzonden: vr 12-2-2010 17:51
Aan: <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>
Onderwerp: [syslog-ng] syslog-ng performance tuning
Hi everybody,
I'have an issue with syslog-ng configuration.
I would like to centralize my logs on one server.
I've a lot of logs to send. I don't know how many but I can estimate it
to 500GB per day from decades of servers.
But, it writes only 25 GB per day.
For some reasons I work on a debian etchnhalf environnement.
So, I'm working with syslog-ng 2.0.0.
I wrote a perl program which spawn p "logger -p <a href="http://local5.info" target="_blank">local5.info</a>" processes
and send n lines of m characters.
I'have tested with:
p: 1 000
n: 1 000
m: 1 000
Instead of having 1 000 000 lines in my logs I have nearly 10 000 lines !
But my test was not revelant because normal logs where not stopped. So,
maybe normal.
</pre>
<pre><hr size="4" width="90%">
______________________________________________________________________________
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
<br>
<div>-- <br>
<div> <span>Rémi BUISSON</span> - <span>IT Engineer</span> <span>F-Secure Storage
& Digital Content</span> <span>7, rue Raymond
Manaud<br>
33524 BORDEAUX Bruges Cedex<br>
FRANCE</span> <img src="cid:part1.07090402.03030004@steek.com" alt="http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png">
</div>
</div>
<pre><hr size="4" width="90%">
______________________________________________________________________________
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
<br>
<div>-- <br>
<div> <span>Rémi BUISSON</span> - <span>IT Engineer</span> <span>F-Secure Storage
& Digital Content</span> <span>7, rue Raymond
Manaud<br>
33524 BORDEAUX Bruges Cedex<br>
FRANCE</span> <img src="cid:part2.09050305.04060004@steek.com" alt="http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png">
</div>
</div>
<pre><hr size="4" width="90%">
______________________________________________________________________________
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
<br>
<div>-- <br>
<div> <span>Rémi BUISSON</span> - <span>IT Engineer</span> <span>F-Secure Storage
& Digital Content</span> <span>7, rue Raymond
Manaud<br>
33524 BORDEAUX Bruges Cedex<br>
FRANCE</span> <img src="cid:part3.06010106.02050301@steek.com" alt="http://www.f-secure.com/export/system/modules/com.fsecure.frontend.newbrand/resources/css/_ui/images/f-secure.png">
</div>
</div>
</div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br></blockquote></div><br>