[syslog-ng] Tests using loggen - not receiving all the packets
Clayton Dukes
cdukes at gmail.com
Wed Apr 14 20:19:25 CEST 2010
Nope - tests still work when using netcat.
On Wed, Apr 14, 2010 at 2:00 PM, Zoltán Pallagi <pzolee at balabit.hu> wrote:
> 2010.04.14. 19:43 keltezéssel, Clayton Dukes írta:
>>
>> Well...now that the system is getting all the messages, it seems that
>> syslog-ng is still not writing everything.
>>
>> #/www/svn/lgentest.sh 10000 10
>> average rate = 10883.79 msg/sec, count=108838, time=10.000, msg
>> size=256, bandwidth=2720.95 kB/sec
>>
>> # wc -l syslog.log
>> 35179 syslog.log
>>
>>
>> Here are my options in the syslog-ng config:
>> options {
>> long_hostnames(off);
>> log_msg_size(8192);
>> flush_lines(1);
>> log_fifo_size(100000);
>> time_reopen(10);
>> use_dns(yes);
>> dns_cache(yes);
>> use_fqdn(yes);
>> keep_hostname(yes);
>> chain_hostnames(no);
>> perm(0644);
>> stats_freq(60);
>>
>> };
>>
>> Any suggestions?
>>
>
>
> Did you restart your machine? The changes under /proc are only temporary
>
>
>>
>> On Wed, Apr 14, 2010 at 12:18 PM, Clayton Dukes<cdukes at gmail.com> wrote:
>>
>>>
>>> For anyone searching the Goog and finding this thread later on, I've
>>> created an explanation of everything in my Wiki:
>>> http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers
>>>
>>> Hope it helps!
>>>
>>>
>>> On Wed, Apr 14, 2010 at 12:10 PM, Clayton Dukes<cdukes at gmail.com> wrote:
>>>
>>>>
>>>> Yay! That did it. Thanks!
>>>>
>>>>
>>>> On Wed, Apr 14, 2010 at 11:30 AM, Zoltán Pallagi<pzolee at balabit.hu>
>>>> wrote:
>>>>
>>>>>
>>>>> Clayton Dukes wrote:
>>>>>
>>>>> Excellent link, thanks!
>>>>> That does seem to be the problem, however, if I set the buffer all the
>>>>> way up to 1G using:
>>>>> sysctl -w net.core.rmem_max=1073741824
>>>>>
>>>>>
>>>>> Then I'm still dropping messages when using a test rate of 6kmps:
>>>>>
>>>>> # ./loggen -r 6000 -D -I 10 127.0.0.1 514
>>>>> average rate = 6526.63 msg/sec, count=65272, time=10.008, msg
>>>>> size=256, bandwidth=1631.66 kB/sec
>>>>>
>>>>> # wc -l /tmp/logs
>>>>> 62933 /tmp/logs
>>>>>
>>>>> Is there a recommendation on what the buffer should be set to for high
>>>>> insertion rates?
>>>>> My test server has 8G of memory, but I can give it more (up to 24G).
>>>>>
>>>>> Also, note that this is a VMWare ESXi server - might that have
>>>>> something to do with it?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> It's interesting. I tried it with rmem_max=1MB, and worked without
>>>>> dropped
>>>>> messages (my machine is Intel(R) Core(TM)2 CPU 6300 @
>>>>> 1.86GHz,
>>>>> with ubuntu)
>>>>>
>>>>> root at thor:/opt/syslog-ng/bin# ./loggen -r 6000 -V -D -I 30 127.0.0.1
>>>>> 2222
>>>>> average rate = 5991.87 msg/sec, count=179757, time=30.001, (last) msg
>>>>> size=256, bandwidth=1497.97 kB/sec
>>>>>
>>>>> root at thor:/var/log# wc -l test.log
>>>>> 179757 test.log
>>>>>
>>>>> root at thor:/var/log# cat /proc/sys/net/core/rmem_default
>>>>> 1048576
>>>>>
>>>>> But if I set the rmem_max to 1MB, I have also dropped packages, If I
>>>>> set the
>>>>> rmem_default it works... (I don't know why, I am not an udp-kernel
>>>>> magus).
>>>>> Will you try if you set rmem_default instead of rmem_max?
>>>>>
>>>>>
>>>>> On Wed, Apr 14, 2010 at 6:16 AM, Zoltán Pallagi<pzolee at balabit.hu>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I think it's not a syslog-ng problem, the udp buffer of your kernel
>>>>> will be
>>>>> full, and the kernel drops the udp packages (to make sure, you can try
>>>>> to
>>>>> use netcat (netcat -lu -p 514>> aaa.txt) instead of syslog-ng, I think
>>>>> the
>>>>> logs will be missed in this case too).
>>>>>
>>>>> before running loggen, please check the value of the packet receive
>>>>> errors:
>>>>> root at thor:/var/log# netstat -su
>>>>> Udp:
>>>>> 124383 packets received
>>>>> 3 packets to unknown port received.
>>>>> 82487 packet receive errors
>>>>> 166196 packets sent
>>>>> RcvbufErrors: 82487
>>>>>
>>>>> then check it after running. I guess, you will see the missing packets
>>>>> (just
>>>>> check the difference between before and after).
>>>>>
>>>>> so, if I am right, you just have to increase the size of the udp
>>>>> receive
>>>>> buffer and it will work.
>>>>> For example:
>>>>> echo "88888888"> /proc/sys/net/core/rmem_default (or rmem_max)
>>>>>
>>>>> for more details about udp buffering:
>>>>> http://www.29west.com/docs/THPM/udp-buffer-sizing.html
>>>>>
>>>>>
>>>>> Clayton Dukes wrote:
>>>>>
>>>>> Finally getting a chance to revisit this.
>>>>> I'm still seeing the problem.
>>>>>
>>>>> If I run loggen like so:
>>>>> /www/svn/loggen -r 600 -D -I 30 127.0.0.1 514
>>>>> average rate = 607.51 msg/sec, count=18226, time=30.012, msg size=256,
>>>>> bandwidth=151.88 kB/sec
>>>>>
>>>>> I only get around 8k messages:
>>>>> wc -l /var/log/logzilla/syslog.log
>>>>> 8740 /var/log/logzilla/syslog.log
>>>>>
>>>>>
>>>>> I've tried bumping up flush_lines and the fifo but neither seemed to
>>>>> make much of a difference.
>>>>>
>>>>> Here's my config:
>>>>> options {
>>>>> long_hostnames(off);
>>>>> log_msg_size(8192);
>>>>> flush_lines(1); # Note: I've tried this up to 1000
>>>>> log_fifo_size(35535);
>>>>> time_reopen(10);
>>>>> use_dns(yes);
>>>>> dns_cache(yes);
>>>>> use_fqdn(yes);
>>>>> keep_hostname(yes);
>>>>> chain_hostnames(no);
>>>>> };
>>>>>
>>>>> destination df_logzilla {
>>>>> file("/var/log/logzilla/syslog.log"
>>>>>
>>>>>
>>>>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>>>>> );
>>>>> };
>>>>>
>>>>> log {
>>>>> source(s_all);
>>>>> destination(df_logzilla);
>>>>> };
>>>>> On Thu, Apr 1, 2010 at 9:33 AM, Martin Holste<mcholste at gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> What do you get if you send the loggen data to a simple netcat session
>>>>> with
>>>>> its output redirected to a flat file? Do you see all 55k messages
>>>>> using wc
>>>>> -l?
>>>>>
>>>>> On Thu, Apr 1, 2010 at 6:51 AM, Clayton Dukes<cdukes at gmail.com> wrote:
>>>>>
>>>>>
>>>>> I should have mentioned that this is logging directly to a file.
>>>>>
>>>>> destination df_logzilla {
>>>>> file("/var/log/logzilla/syslog.log"
>>>>>
>>>>>
>>>>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>>>>> );
>>>>> };
>>>>>
>>>>>
>>>>> On Wed, Mar 31, 2010 at 11:47 PM, Clayton Dukes<cdukes at gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hi Folks,
>>>>> I'm trying to run a test to check insert rates.
>>>>> If I run this command:
>>>>>
>>>>> ./loggen -r 5000 -D -I 10 127.0.0.1 514
>>>>>
>>>>> The output shows:
>>>>> average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256,
>>>>> bandwidth=1360.40 kB/sec
>>>>>
>>>>> But, my stats don't show that many messages received:
>>>>>
>>>>> syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\',
>>>>> processed=\'center(queued)=24232\', processed=\'center(received)=8077,
>>>>> processed=\'destination(df_logzilla)=8077\'
>>>>>
>>>>> As you can see, it sent 55k messages, but I only received 8k.
>>>>> Am I doing something wrong?
>>>>>
>>>>> Here are my options in the syslog-ng config:
>>>>> options {
>>>>> long_hostnames(off);
>>>>> log_msg_size(8192);
>>>>> flush_lines(1);
>>>>> log_fifo_size(16384);
>>>>> time_reopen(10);
>>>>> use_dns(yes);
>>>>> dns_cache(yes);
>>>>> use_fqdn(yes);
>>>>> keep_hostname(yes);
>>>>> chain_hostnames(no);
>>>>> perm(0644);
>>>>> stats_freq(60);
>>>>>
>>>>> };
>>>>>
>>>>>
>>>>> --
>>>>> ______________________________________________________________
>>>>>
>>>>> Clayton Dukes
>>>>> ______________________________________________________________
>>>>>
>>>>>
>>>>> --
>>>>> ______________________________________________________________
>>>>>
>>>>> Clayton Dukes
>>>>> ______________________________________________________________
>>>>>
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> pzolee
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> pzolee
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> ______________________________________________________________
>>>>
>>>> Clayton Dukes
>>>> ______________________________________________________________
>>>>
>>>>
>>>
>>>
>>> --
>>> ______________________________________________________________
>>>
>>> Clayton Dukes
>>> ______________________________________________________________
>>>
>>>
>>
>>
>>
>
>
> --
> pzolee
>
--
______________________________________________________________
Clayton Dukes
______________________________________________________________
More information about the syslog-ng
mailing list