[syslog-ng] Tests using loggen - not receiving all the packets
Clayton Dukes
cdukes at gmail.com
Wed Apr 14 18:18:33 CEST 2010
For anyone searching the Goog and finding this thread later on, I've
created an explanation of everything in my Wiki:
http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers
Hope it helps!
On Wed, Apr 14, 2010 at 12:10 PM, Clayton Dukes <cdukes at gmail.com> wrote:
> Yay! That did it. Thanks!
>
>
> On Wed, Apr 14, 2010 at 11:30 AM, Zoltán Pallagi <pzolee at balabit.hu> wrote:
>> Clayton Dukes wrote:
>>
>> Excellent link, thanks!
>> That does seem to be the problem, however, if I set the buffer all the
>> way up to 1G using:
>> sysctl -w net.core.rmem_max=1073741824
>>
>>
>> Then I'm still dropping messages when using a test rate of 6kmps:
>>
>> # ./loggen -r 6000 -D -I 10 127.0.0.1 514
>> average rate = 6526.63 msg/sec, count=65272, time=10.008, msg
>> size=256, bandwidth=1631.66 kB/sec
>>
>> # wc -l /tmp/logs
>> 62933 /tmp/logs
>>
>> Is there a recommendation on what the buffer should be set to for high
>> insertion rates?
>> My test server has 8G of memory, but I can give it more (up to 24G).
>>
>> Also, note that this is a VMWare ESXi server - might that have
>> something to do with it?
>>
>>
>>
>>
>> It's interesting. I tried it with rmem_max=1MB, and worked without dropped
>> messages (my machine is Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz,
>> with ubuntu)
>>
>> root at thor:/opt/syslog-ng/bin# ./loggen -r 6000 -V -D -I 30 127.0.0.1 2222
>> average rate = 5991.87 msg/sec, count=179757, time=30.001, (last) msg
>> size=256, bandwidth=1497.97 kB/sec
>>
>> root at thor:/var/log# wc -l test.log
>> 179757 test.log
>>
>> root at thor:/var/log# cat /proc/sys/net/core/rmem_default
>> 1048576
>>
>> But if I set the rmem_max to 1MB, I have also dropped packages, If I set the
>> rmem_default it works... (I don't know why, I am not an udp-kernel magus).
>> Will you try if you set rmem_default instead of rmem_max?
>>
>>
>> On Wed, Apr 14, 2010 at 6:16 AM, Zoltán Pallagi <pzolee at balabit.hu> wrote:
>>
>>
>> Hi,
>>
>> I think it's not a syslog-ng problem, the udp buffer of your kernel will be
>> full, and the kernel drops the udp packages (to make sure, you can try to
>> use netcat (netcat -lu -p 514 >> aaa.txt) instead of syslog-ng, I think the
>> logs will be missed in this case too).
>>
>> before running loggen, please check the value of the packet receive errors:
>> root at thor:/var/log# netstat -su
>> Udp:
>> 124383 packets received
>> 3 packets to unknown port received.
>> 82487 packet receive errors
>> 166196 packets sent
>> RcvbufErrors: 82487
>>
>> then check it after running. I guess, you will see the missing packets (just
>> check the difference between before and after).
>>
>> so, if I am right, you just have to increase the size of the udp receive
>> buffer and it will work.
>> For example:
>> echo "88888888" > /proc/sys/net/core/rmem_default (or rmem_max)
>>
>> for more details about udp buffering:
>> http://www.29west.com/docs/THPM/udp-buffer-sizing.html
>>
>>
>> Clayton Dukes wrote:
>>
>> Finally getting a chance to revisit this.
>> I'm still seeing the problem.
>>
>> If I run loggen like so:
>> /www/svn/loggen -r 600 -D -I 30 127.0.0.1 514
>> average rate = 607.51 msg/sec, count=18226, time=30.012, msg size=256,
>> bandwidth=151.88 kB/sec
>>
>> I only get around 8k messages:
>> wc -l /var/log/logzilla/syslog.log
>> 8740 /var/log/logzilla/syslog.log
>>
>>
>> I've tried bumping up flush_lines and the fifo but neither seemed to
>> make much of a difference.
>>
>> Here's my config:
>> options {
>> long_hostnames(off);
>> log_msg_size(8192);
>> flush_lines(1); # Note: I've tried this up to 1000
>> log_fifo_size(35535);
>> time_reopen(10);
>> use_dns(yes);
>> dns_cache(yes);
>> use_fqdn(yes);
>> keep_hostname(yes);
>> chain_hostnames(no);
>> };
>>
>> destination df_logzilla {
>> file("/var/log/logzilla/syslog.log"
>>
>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>> );
>> };
>>
>> log {
>> source(s_all);
>> destination(df_logzilla);
>> };
>> On Thu, Apr 1, 2010 at 9:33 AM, Martin Holste <mcholste at gmail.com> wrote:
>>
>>
>> What do you get if you send the loggen data to a simple netcat session with
>> its output redirected to a flat file? Do you see all 55k messages using wc
>> -l?
>>
>> On Thu, Apr 1, 2010 at 6:51 AM, Clayton Dukes <cdukes at gmail.com> wrote:
>>
>>
>> I should have mentioned that this is logging directly to a file.
>>
>> destination df_logzilla {
>> file("/var/log/logzilla/syslog.log"
>>
>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>> );
>> };
>>
>>
>> On Wed, Mar 31, 2010 at 11:47 PM, Clayton Dukes <cdukes at gmail.com> wrote:
>>
>>
>> Hi Folks,
>> I'm trying to run a test to check insert rates.
>> If I run this command:
>>
>> ./loggen -r 5000 -D -I 10 127.0.0.1 514
>>
>> The output shows:
>> average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256,
>> bandwidth=1360.40 kB/sec
>>
>> But, my stats don't show that many messages received:
>>
>> syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\',
>> processed=\'center(queued)=24232\', processed=\'center(received)=8077,
>> processed=\'destination(df_logzilla)=8077\'
>>
>> As you can see, it sent 55k messages, but I only received 8k.
>> Am I doing something wrong?
>>
>> Here are my options in the syslog-ng config:
>> options {
>> long_hostnames(off);
>> log_msg_size(8192);
>> flush_lines(1);
>> log_fifo_size(16384);
>> time_reopen(10);
>> use_dns(yes);
>> dns_cache(yes);
>> use_fqdn(yes);
>> keep_hostname(yes);
>> chain_hostnames(no);
>> perm(0644);
>> stats_freq(60);
>>
>> };
>>
>>
>> --
>> ______________________________________________________________
>>
>> Clayton Dukes
>> ______________________________________________________________
>>
>>
>> --
>> ______________________________________________________________
>>
>> Clayton Dukes
>> ______________________________________________________________
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> pzolee
>>
>>
>>
>>
>>
>> --
>> pzolee
>>
>
>
>
> --
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________
>
--
______________________________________________________________
Clayton Dukes
______________________________________________________________
More information about the syslog-ng
mailing list