[syslog-ng] Tests using loggen - not receiving all the packets

Clayton Dukes cdukes at gmail.com
Wed Apr 14 19:43:54 CEST 2010


Well...now that the system is getting all the messages, it seems that
syslog-ng is still not writing everything.

#/www/svn/lgentest.sh 10000 10
average rate = 10883.79 msg/sec, count=108838, time=10.000, msg
size=256, bandwidth=2720.95 kB/sec

# wc -l syslog.log
35179 syslog.log


Here are my options in the syslog-ng config:
options {
      long_hostnames(off);
      log_msg_size(8192);
      flush_lines(1);
      log_fifo_size(100000);
      time_reopen(10);
      use_dns(yes);
      dns_cache(yes);
      use_fqdn(yes);
      keep_hostname(yes);
      chain_hostnames(no);
      perm(0644);
     stats_freq(60);

};

Any suggestions?


On Wed, Apr 14, 2010 at 12:18 PM, Clayton Dukes <cdukes at gmail.com> wrote:
> For anyone searching the Goog and finding this thread later on, I've
> created an explanation of everything in my Wiki:
> http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers
>
> Hope it helps!
>
>
> On Wed, Apr 14, 2010 at 12:10 PM, Clayton Dukes <cdukes at gmail.com> wrote:
>> Yay! That did it. Thanks!
>>
>>
>> On Wed, Apr 14, 2010 at 11:30 AM, Zoltán Pallagi <pzolee at balabit.hu> wrote:
>>> Clayton Dukes wrote:
>>>
>>> Excellent link, thanks!
>>> That does seem to be the problem, however, if I set the buffer all the
>>> way up to 1G using:
>>> sysctl -w net.core.rmem_max=1073741824
>>>
>>>
>>> Then I'm still dropping messages when using a test rate of 6kmps:
>>>
>>> # ./loggen -r 6000 -D -I 10 127.0.0.1 514
>>> average rate = 6526.63 msg/sec, count=65272, time=10.008, msg
>>> size=256, bandwidth=1631.66 kB/sec
>>>
>>> # wc -l /tmp/logs
>>> 62933 /tmp/logs
>>>
>>> Is there a recommendation on what the buffer should be set to for high
>>> insertion rates?
>>> My test server has 8G of memory, but I can give it more (up to 24G).
>>>
>>> Also, note that this is a VMWare ESXi server - might that have
>>> something to do with it?
>>>
>>>
>>>
>>>
>>> It's interesting. I tried it with rmem_max=1MB, and worked without dropped
>>> messages (my machine is  Intel(R) Core(TM)2 CPU          6300  @ 1.86GHz,
>>> with ubuntu)
>>>
>>> root at thor:/opt/syslog-ng/bin# ./loggen -r 6000 -V -D -I 30 127.0.0.1 2222
>>> average rate = 5991.87 msg/sec, count=179757, time=30.001, (last) msg
>>> size=256, bandwidth=1497.97 kB/sec
>>>
>>> root at thor:/var/log# wc -l test.log
>>> 179757 test.log
>>>
>>> root at thor:/var/log# cat /proc/sys/net/core/rmem_default
>>> 1048576
>>>
>>> But if I set the rmem_max to 1MB, I have also dropped packages, If I set the
>>> rmem_default it works... (I don't know why, I am not an udp-kernel magus).
>>> Will you try if you set rmem_default instead of rmem_max?
>>>
>>>
>>> On Wed, Apr 14, 2010 at 6:16 AM, Zoltán Pallagi <pzolee at balabit.hu> wrote:
>>>
>>>
>>> Hi,
>>>
>>> I think it's not a syslog-ng problem, the udp buffer of your kernel will be
>>> full, and the kernel drops the udp packages (to make sure, you can try to
>>> use netcat (netcat -lu -p 514 >> aaa.txt) instead of syslog-ng, I think the
>>> logs will be missed in this case too).
>>>
>>> before running loggen, please check the value of the packet receive errors:
>>> root at thor:/var/log# netstat -su
>>> Udp:
>>>     124383 packets received
>>>     3 packets to unknown port received.
>>>     82487 packet receive errors
>>>     166196 packets sent
>>>     RcvbufErrors: 82487
>>>
>>> then check it after running. I guess, you will see the missing packets (just
>>> check the difference between before and after).
>>>
>>> so, if I am right, you just have to increase the size of the udp receive
>>> buffer and it will work.
>>> For example:
>>> echo "88888888" > /proc/sys/net/core/rmem_default (or rmem_max)
>>>
>>> for more details about udp buffering:
>>> http://www.29west.com/docs/THPM/udp-buffer-sizing.html
>>>
>>>
>>> Clayton Dukes wrote:
>>>
>>> Finally getting a chance to revisit this.
>>> I'm still seeing the problem.
>>>
>>> If I run loggen like so:
>>> /www/svn/loggen -r 600 -D -I 30 127.0.0.1 514
>>> average rate = 607.51 msg/sec, count=18226, time=30.012, msg size=256,
>>> bandwidth=151.88 kB/sec
>>>
>>> I only get around 8k messages:
>>> wc -l /var/log/logzilla/syslog.log
>>> 8740 /var/log/logzilla/syslog.log
>>>
>>>
>>> I've tried bumping up flush_lines and the fifo but neither seemed to
>>> make much of a difference.
>>>
>>> Here's my config:
>>> options {
>>>       long_hostnames(off);
>>>       log_msg_size(8192);
>>>       flush_lines(1); # Note: I've tried this up to 1000
>>>       log_fifo_size(35535);
>>>       time_reopen(10);
>>>       use_dns(yes);
>>>       dns_cache(yes);
>>>       use_fqdn(yes);
>>>       keep_hostname(yes);
>>>       chain_hostnames(no);
>>> };
>>>
>>> destination df_logzilla {
>>>    file("/var/log/logzilla/syslog.log"
>>>
>>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>>>    );
>>> };
>>>
>>> log {
>>>    source(s_all);
>>>       destination(df_logzilla);
>>> };
>>> On Thu, Apr 1, 2010 at 9:33 AM, Martin Holste <mcholste at gmail.com> wrote:
>>>
>>>
>>> What do you get if you send the loggen data to a simple netcat session with
>>> its output redirected to a flat file?  Do you see all 55k messages using wc
>>> -l?
>>>
>>> On Thu, Apr 1, 2010 at 6:51 AM, Clayton Dukes <cdukes at gmail.com> wrote:
>>>
>>>
>>> I should have mentioned that this is logging directly to a file.
>>>
>>> destination df_logzilla {
>>>    file("/var/log/logzilla/syslog.log"
>>>
>>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>>>    );
>>> };
>>>
>>>
>>> On Wed, Mar 31, 2010 at 11:47 PM, Clayton Dukes <cdukes at gmail.com> wrote:
>>>
>>>
>>> Hi Folks,
>>> I'm trying to run a test to check insert rates.
>>> If I run this command:
>>>
>>> ./loggen -r 5000 -D -I 10 127.0.0.1 514
>>>
>>> The output shows:
>>> average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256,
>>> bandwidth=1360.40 kB/sec
>>>
>>> But, my stats don't show that many messages received:
>>>
>>> syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\',
>>> processed=\'center(queued)=24232\', processed=\'center(received)=8077,
>>> processed=\'destination(df_logzilla)=8077\'
>>>
>>> As you can see, it sent 55k messages, but I only received 8k.
>>> Am I doing something wrong?
>>>
>>> Here are my options in the syslog-ng config:
>>> options {
>>>       long_hostnames(off);
>>>       log_msg_size(8192);
>>>       flush_lines(1);
>>>       log_fifo_size(16384);
>>>       time_reopen(10);
>>>       use_dns(yes);
>>>       dns_cache(yes);
>>>       use_fqdn(yes);
>>>       keep_hostname(yes);
>>>       chain_hostnames(no);
>>>       perm(0644);
>>>      stats_freq(60);
>>>
>>> };
>>>
>>>
>>> --
>>> ______________________________________________________________
>>>
>>> Clayton Dukes
>>> ______________________________________________________________
>>>
>>>
>>> --
>>> ______________________________________________________________
>>>
>>> Clayton Dukes
>>> ______________________________________________________________
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> pzolee
>>>
>>>
>>>
>>>
>>>
>>> --
>>> pzolee
>>>
>>
>>
>>
>> --
>> ______________________________________________________________
>>
>> Clayton Dukes
>> ______________________________________________________________
>>
>
>
>
> --
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________
>



-- 
______________________________________________________________

Clayton Dukes
______________________________________________________________


More information about the syslog-ng mailing list