[syslog-ng] Tests using loggen - not receiving all the packets

Zoltán Pallagi pzolee at balabit.hu
Wed Apr 14 12:16:59 CEST 2010


Hi,

I think it's not a syslog-ng problem, the udp buffer of your kernel will 
be full, and the kernel drops the udp packages (to make sure, you can 
try to use netcat (netcat -lu -p 514 >> aaa.txt) instead of syslog-ng, I 
think the logs will be missed in this case too).

before running loggen, please check the value of the packet receive errors:
root at thor:/var/log# netstat -su
Udp:
    124383 packets received
    3 packets to unknown port received.
    *82487 packet receive errors*
    166196 packets sent
    RcvbufErrors: 82487

then check it after running. I guess, you will see the missing packets 
(just check the difference between before and after).

so, if I am right, you just have to increase the size of the udp receive 
buffer and it will work.
For example:
echo "88888888" > /proc/sys/net/core/rmem_default (or rmem_max)

for more details about udp buffering:
http://www.29west.com/docs/THPM/udp-buffer-sizing.html


Clayton Dukes wrote:
> Finally getting a chance to revisit this.
> I'm still seeing the problem.
>
> If I run loggen like so:
> /www/svn/loggen -r 600 -D -I 30 127.0.0.1 514
> average rate = 607.51 msg/sec, count=18226, time=30.012, msg size=256,
> bandwidth=151.88 kB/sec
>
> I only get around 8k messages:
> wc -l /var/log/logzilla/syslog.log
> 8740 /var/log/logzilla/syslog.log
>
>
> I've tried bumping up flush_lines and the fifo but neither seemed to
> make much of a difference.
>
> Here's my config:
> options {
>       long_hostnames(off);
>       log_msg_size(8192);
>       flush_lines(1); # Note: I've tried this up to 1000
>       log_fifo_size(35535);
>       time_reopen(10);
>       use_dns(yes);
>       dns_cache(yes);
>       use_fqdn(yes);
>       keep_hostname(yes);
>       chain_hostnames(no);
> };
>
> destination df_logzilla {
>    file("/var/log/logzilla/syslog.log"
>    template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>    );
> };
>
> log {
>    source(s_all);
>       destination(df_logzilla);
> };
> On Thu, Apr 1, 2010 at 9:33 AM, Martin Holste <mcholste at gmail.com> wrote:
>   
>> What do you get if you send the loggen data to a simple netcat session with
>> its output redirected to a flat file?  Do you see all 55k messages using wc
>> -l?
>>
>> On Thu, Apr 1, 2010 at 6:51 AM, Clayton Dukes <cdukes at gmail.com> wrote:
>>     
>>> I should have mentioned that this is logging directly to a file.
>>>
>>> destination df_logzilla {
>>>    file("/var/log/logzilla/syslog.log"
>>>
>>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>>>    );
>>> };
>>>
>>>
>>> On Wed, Mar 31, 2010 at 11:47 PM, Clayton Dukes <cdukes at gmail.com> wrote:
>>>       
>>>> Hi Folks,
>>>> I'm trying to run a test to check insert rates.
>>>> If I run this command:
>>>>
>>>> ./loggen -r 5000 -D -I 10 127.0.0.1 514
>>>>
>>>> The output shows:
>>>> average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256,
>>>> bandwidth=1360.40 kB/sec
>>>>
>>>> But, my stats don't show that many messages received:
>>>>
>>>> syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\',
>>>> processed=\'center(queued)=24232\', processed=\'center(received)=8077,
>>>> processed=\'destination(df_logzilla)=8077\'
>>>>
>>>> As you can see, it sent 55k messages, but I only received 8k.
>>>> Am I doing something wrong?
>>>>
>>>> Here are my options in the syslog-ng config:
>>>> options {
>>>>       long_hostnames(off);
>>>>       log_msg_size(8192);
>>>>       flush_lines(1);
>>>>       log_fifo_size(16384);
>>>>       time_reopen(10);
>>>>       use_dns(yes);
>>>>       dns_cache(yes);
>>>>       use_fqdn(yes);
>>>>       keep_hostname(yes);
>>>>       chain_hostnames(no);
>>>>       perm(0644);
>>>>      stats_freq(60);
>>>>
>>>> };
>>>>
>>>>
>>>> --
>>>> ______________________________________________________________
>>>>
>>>> Clayton Dukes
>>>> ______________________________________________________________
>>>>         
>>>
>>> --
>>> ______________________________________________________________
>>>
>>> Clayton Dukes
>>> ______________________________________________________________
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>
>>>       
>>     
>
>
>
>   


-- 
pzolee

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100414/be3f2d55/attachment.htm 


More information about the syslog-ng mailing list