[syslog-ng] Tests using loggen - not receiving all the packets
Clayton Dukes
cdukes at gmail.com
Wed Apr 14 18:10:36 CEST 2010
Yay! That did it. Thanks!
On Wed, Apr 14, 2010 at 11:30 AM, Zoltán Pallagi <pzolee at balabit.hu> wrote:
> Clayton Dukes wrote:
>
> Excellent link, thanks!
> That does seem to be the problem, however, if I set the buffer all the
> way up to 1G using:
> sysctl -w net.core.rmem_max=1073741824
>
>
> Then I'm still dropping messages when using a test rate of 6kmps:
>
> # ./loggen -r 6000 -D -I 10 127.0.0.1 514
> average rate = 6526.63 msg/sec, count=65272, time=10.008, msg
> size=256, bandwidth=1631.66 kB/sec
>
> # wc -l /tmp/logs
> 62933 /tmp/logs
>
> Is there a recommendation on what the buffer should be set to for high
> insertion rates?
> My test server has 8G of memory, but I can give it more (up to 24G).
>
> Also, note that this is a VMWare ESXi server - might that have
> something to do with it?
>
>
>
>
> It's interesting. I tried it with rmem_max=1MB, and worked without dropped
> messages (my machine is Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz,
> with ubuntu)
>
> root at thor:/opt/syslog-ng/bin# ./loggen -r 6000 -V -D -I 30 127.0.0.1 2222
> average rate = 5991.87 msg/sec, count=179757, time=30.001, (last) msg
> size=256, bandwidth=1497.97 kB/sec
>
> root at thor:/var/log# wc -l test.log
> 179757 test.log
>
> root at thor:/var/log# cat /proc/sys/net/core/rmem_default
> 1048576
>
> But if I set the rmem_max to 1MB, I have also dropped packages, If I set the
> rmem_default it works... (I don't know why, I am not an udp-kernel magus).
> Will you try if you set rmem_default instead of rmem_max?
>
>
> On Wed, Apr 14, 2010 at 6:16 AM, Zoltán Pallagi <pzolee at balabit.hu> wrote:
>
>
> Hi,
>
> I think it's not a syslog-ng problem, the udp buffer of your kernel will be
> full, and the kernel drops the udp packages (to make sure, you can try to
> use netcat (netcat -lu -p 514 >> aaa.txt) instead of syslog-ng, I think the
> logs will be missed in this case too).
>
> before running loggen, please check the value of the packet receive errors:
> root at thor:/var/log# netstat -su
> Udp:
> 124383 packets received
> 3 packets to unknown port received.
> 82487 packet receive errors
> 166196 packets sent
> RcvbufErrors: 82487
>
> then check it after running. I guess, you will see the missing packets (just
> check the difference between before and after).
>
> so, if I am right, you just have to increase the size of the udp receive
> buffer and it will work.
> For example:
> echo "88888888" > /proc/sys/net/core/rmem_default (or rmem_max)
>
> for more details about udp buffering:
> http://www.29west.com/docs/THPM/udp-buffer-sizing.html
>
>
> Clayton Dukes wrote:
>
> Finally getting a chance to revisit this.
> I'm still seeing the problem.
>
> If I run loggen like so:
> /www/svn/loggen -r 600 -D -I 30 127.0.0.1 514
> average rate = 607.51 msg/sec, count=18226, time=30.012, msg size=256,
> bandwidth=151.88 kB/sec
>
> I only get around 8k messages:
> wc -l /var/log/logzilla/syslog.log
> 8740 /var/log/logzilla/syslog.log
>
>
> I've tried bumping up flush_lines and the fifo but neither seemed to
> make much of a difference.
>
> Here's my config:
> options {
> long_hostnames(off);
> log_msg_size(8192);
> flush_lines(1); # Note: I've tried this up to 1000
> log_fifo_size(35535);
> time_reopen(10);
> use_dns(yes);
> dns_cache(yes);
> use_fqdn(yes);
> keep_hostname(yes);
> chain_hostnames(no);
> };
>
> destination df_logzilla {
> file("/var/log/logzilla/syslog.log"
>
> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
> );
> };
>
> log {
> source(s_all);
> destination(df_logzilla);
> };
> On Thu, Apr 1, 2010 at 9:33 AM, Martin Holste <mcholste at gmail.com> wrote:
>
>
> What do you get if you send the loggen data to a simple netcat session with
> its output redirected to a flat file? Do you see all 55k messages using wc
> -l?
>
> On Thu, Apr 1, 2010 at 6:51 AM, Clayton Dukes <cdukes at gmail.com> wrote:
>
>
> I should have mentioned that this is logging directly to a file.
>
> destination df_logzilla {
> file("/var/log/logzilla/syslog.log"
>
> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
> );
> };
>
>
> On Wed, Mar 31, 2010 at 11:47 PM, Clayton Dukes <cdukes at gmail.com> wrote:
>
>
> Hi Folks,
> I'm trying to run a test to check insert rates.
> If I run this command:
>
> ./loggen -r 5000 -D -I 10 127.0.0.1 514
>
> The output shows:
> average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256,
> bandwidth=1360.40 kB/sec
>
> But, my stats don't show that many messages received:
>
> syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\',
> processed=\'center(queued)=24232\', processed=\'center(received)=8077,
> processed=\'destination(df_logzilla)=8077\'
>
> As you can see, it sent 55k messages, but I only received 8k.
> Am I doing something wrong?
>
> Here are my options in the syslog-ng config:
> options {
> long_hostnames(off);
> log_msg_size(8192);
> flush_lines(1);
> log_fifo_size(16384);
> time_reopen(10);
> use_dns(yes);
> dns_cache(yes);
> use_fqdn(yes);
> keep_hostname(yes);
> chain_hostnames(no);
> perm(0644);
> stats_freq(60);
>
> };
>
>
> --
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________
>
>
> --
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>
>
>
>
>
>
> --
> pzolee
>
>
>
>
>
> --
> pzolee
>
--
______________________________________________________________
Clayton Dukes
______________________________________________________________
More information about the syslog-ng
mailing list