[syslog-ng] Tests using loggen - not receiving all the packets
Patrick H.
syslogng at feystorm.net
Thu Apr 1 18:37:31 CEST 2010
Nevermind, I found my problem. Flush lines was set to 10, and as I've
got about 500 output files open, it was overflowing the input buffer, so
it thought it needed to turn on flow-control. Had to bump up log_iw_size.
Sent: Thursday, April 01, 2010 9:36:22 AM
From: Patrick H. <syslogng at feystorm.net>
To: Syslog-ng users' and developers' mailing list
<syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Tests using loggen - not receiving all the packets
> I think I'm getting this behavior as well. I use a master syslog-ng
> process to send filtered packets to another syslog-ng process on the
> same box over tcp, and its going horridly slow. I tried replacing the
> syslog-ng child process with netcat, and it the master process is
> sending really fast. But whenever I start up the child syslog-ng, it
> starts backing up and the master process starts buffering all the data
> it cant send.
> This behavior seems to have started after upgrading to 3.1.0. I'll try
> downgrading back to 3.0.5 and see if it continues, as well as try to
> figure out why its performing so bad.
> I've done an strace, and its definitely having something fishy happen
> with the socket.
>
> (syslog-ng.516 is the process name of the child process)
> # strace -p `pidof syslog-ng.516` -i -r -T
> ...
> 0.000029 [ 370640d510] write(47, "15364 [3845467040] Apr 1
> 15:21:"..., 125) = 125 <0.000009>
> 0.000046 [ 370640dd40] lseek(47, 0, SEEK_END) = 3546874498
> <0.000006>
> 0.000029 [ 370640d510] write(47, "15364 [3845467040] Apr 1
> 15:21:"..., 186) = 186 <0.000009>
> 0.000067 [ 3705ccae0f] poll([{fd=3, events=POLLIN}, {fd=5,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=0}], 4, 0) = 0
> (Timeout) <0.000008>
> 0.000307 [ 3705ccae0f] poll([{fd=3, events=POLLIN}, {fd=5,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=0}], 4, 0) = 0
> (Timeout) <0.000010>
> 0.000310 [ 3705ccae0f] poll([{fd=3, events=POLLIN}, {fd=5,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=0}], 4, 0) = 0
> (Timeout) <0.000008>
> 0.000262 [ 3705ccae0f] poll([{fd=3, events=POLLIN}, {fd=5,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=0}], 4, 4648) = 0
> (Timeout) <4.648231>
> 4.648391 [ 370640dd40] lseek(15, 0, SEEK_END) = 413347569
> <0.000009>
> 0.000041 [ 370640d510] write(15, "Apr 1 15:21:49 gwsinam03
> postfi"..., 230) = 230 <0.000015>
> 0.000074 [ 370640dd40] lseek(15, 0, SEEK_END) = 413347799
> <0.000006>
> 0.000033 [ 370640d510] write(15, "Apr 1 15:21:49 gwsinam03
> postfi"..., 68) = 68 <0.000023>
>
> As you can see, its sitting on that poll for 4 seconds before continuing.
> From lsof, the FDs its polling:
> syslog-ng 29232 root 3r FIFO 0,6
> 214383200 pipe
> syslog-ng 29232 root 5u unix 0xffff81022f7bf680
> 214383203 /var/lib/syslog-ng/syslog-ng.ctl
> syslog-ng 29232 root 7u IPv4 214383201
> TCP 127.0.0.1:516 (LISTEN)
> syslog-ng 29232 root 9u IPv4 214383285
> TCP 127.0.0.1->127.0.0.1:28061 (ESTABLISHED)
>
>
>
> Sent: Thursday, April 01, 2010 7:33:12 AM
> From: Martin Holste <mcholste at gmail.com>
> To: cdukes at cdukes.com, Syslog-ng users' and developers' mailing list
> <syslog-ng at lists.balabit.hu>
> Subject: Re: [syslog-ng] Tests using loggen - not receiving all the
> packets
>> What do you get if you send the loggen data to a simple netcat
>> session with its output redirected to a flat file? Do you see all
>> 55k messages using wc -l?
>>
>> On Thu, Apr 1, 2010 at 6:51 AM, Clayton Dukes <cdukes at gmail.com
>> <mailto:cdukes at gmail.com>> wrote:
>>
>> I should have mentioned that this is logging directly to a file.
>>
>> destination df_logzilla {
>> file("/var/log/logzilla/syslog.log"
>>
>> template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
>> );
>> };
>>
>>
>>
>> On Wed, Mar 31, 2010 at 11:47 PM, Clayton Dukes <cdukes at gmail.com
>> <mailto:cdukes at gmail.com>> wrote:
>>
>> Hi Folks,
>> I'm trying to run a test to check insert rates.
>> If I run this command:
>>
>> ./loggen -r 5000 -D -I 10 127.0.0.1 514
>>
>> The output shows:
>> average rate = 5441.60 msg/sec, count=54420, time=10.007, msg
>> size=256, bandwidth=1360.40 kB/sec
>>
>> But, my stats don't show that many messages received:
>>
>> syslog-ng[6660]: Log statistics;
>> dropped=\'pipe(/dev/xconsole)=0\',
>> processed=\'center(queued)=24232\',
>> processed=\'center(received)=8077,
>> processed=\'destination(df_logzilla)=8077\'
>>
>> As you can see, it sent 55k messages, but I only received 8k.
>> Am I doing something wrong?
>>
>> Here are my options in the syslog-ng config:
>> options {
>> long_hostnames(off);
>> log_msg_size(8192);
>> flush_lines(1);
>> log_fifo_size(16384);
>> time_reopen(10);
>> use_dns(yes);
>> dns_cache(yes);
>> use_fqdn(yes);
>> keep_hostname(yes);
>> chain_hostnames(no);
>> perm(0644);
>> stats_freq(60);
>>
>> };
>>
>>
>> --
>> ______________________________________________________________
>>
>> Clayton Dukes
>> ______________________________________________________________
>>
>>
>>
>>
>> --
>> ______________________________________________________________
>>
>> Clayton Dukes
>> ______________________________________________________________
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
> ------------------------------------------------------------------------
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100401/4dcd1a3a/attachment-0001.htm
More information about the syslog-ng
mailing list