[syslog-ng] Syslog-ng dropped packets.

[Vaibhav Goel]

Yup...

Below are some of the stats.  As you can see, in peak hours, the rate of
the dropped messages is huge.  I have increased the log_fifo_size in the
global options to 4096 (from 2000)

options {
   sync (0);
   time_reopen (10);
   log_fifo_size (4096);
   long_hostnames (on);
   use_dns (yes);
}




Jul 13 15:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
9445
Jul 13 15:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
18944
Jul 13 15:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
10517
Jul 13 15:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
12071
Jul 13 16:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
17505
Jul 13 16:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
12226
Jul 13 16:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
8276
Jul 13 16:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
13717
Jul 13 16:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
14688
Jul 13 16:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
4136
Jul 13 17:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
5506
Jul 13 17:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
2328
Jul 13 17:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1754
Jul 13 17:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
3253
Jul 13 17:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
2449
Jul 13 17:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1023
Jul 13 18:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1868
Jul 13 18:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1541
Jul 13 18:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1863
Jul 13 18:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 380
Jul 13 18:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1340
Jul 13 18:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1241
Jul 13 19:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1693
Jul 13 19:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1858
Jul 13 19:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 13 19:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 651
Jul 13 19:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 739
Jul 13 19:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1894
Jul 13 20:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 294
Jul 13 20:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 13 20:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 586
Jul 13 20:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 996
Jul 13 20:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 203
Jul 13 20:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 594
Jul 13 21:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
2365
Jul 13 21:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1158
Jul 13 21:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
1065
Jul 13 21:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 949
Jul 13 21:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
2341
Jul 13 21:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 98
Jul 13 22:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 382
Jul 13 22:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 50
Jul 13 22:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 84
Jul 13 22:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 293
Jul 13 22:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 54
Jul 13 22:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 351
Jul 13 23:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 6
Jul 13 23:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 493
Jul 13 23:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 13 23:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 13 23:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 13 23:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 00:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 00:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 00:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 00:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 00:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 00:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 01:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 01:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 01:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 01:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 01:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 01:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 02:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 02:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 02:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 02:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 02:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 02:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 03:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 03:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 03:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 03:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 03:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 03:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 04:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 04:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 04:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 04:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 04:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 04:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 05:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 05:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 05:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 05:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 05:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 05:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 06:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 06:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 06:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 06:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 06:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 06:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 623
Jul 14 07:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 07:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped 0
Jul 14 07:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
10755
Jul 14 07:39:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
6718
Jul 14 07:49:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
17484
Jul 14 07:59:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
30382
Jul 14 08:09:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
32052
Jul 14 08:19:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
47914
Jul 14 08:29:55 s_syslogng at clxp01ml syslog-ng[29324]: STATS: dropped
44719
Jul 14 08:41:31 s_syslogng at clxp01ml syslog-ng[8094]: STATS: dropped
37327
Jul 14 08:51:31 s_syslogng at clxp01ml syslog-ng[8094]: STATS: dropped
49299

-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu
[mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs
Scheidler
Sent: July 14, 2006 8:27 AM
To: Syslog-ng users' and developers' mailing list
Subject: RE: [syslog-ng] Syslog-ng dropped packets.

On Fri, 2006-07-14 at 06:21 -0700, Vaibhav Goel wrote:
> Hi Bazsi
> 
> We are still having issues with dropped packets under heavy load.  We
> increased the size of rmem_default and rmem_max drastically but we are
> not seeing much of an improvement....Maybe about 50% or so.
Currently,
> the sizes are 
> 
> $ > cat /proc/sys/net/core/rmem_default
> 786426
> $ > cat /proc/sys/net/core/rmem_max
> 1572852
> 
> Do you have any other suggestions?  We are running syslog-ng 1.6.11.
> 
> All the logging is done over TCP (except 1 UDP source but that doesn't
> generate a lot of logging data).

It would be important to know whether it is syslog-ng itself that drops
messages or something else in the system.

Do you have STATS lines in your logs? That shows how many lines were
dropped by syslog-ng itself. It is also possible that it is the sending
syslog-ng process that drops messages (in the case of TCP), you can
verify this by checking the STATS line of the sending syslog-ng process.
You can improve this situation somewhat by increasing log_fifo_size on
the sender.

1.6.x sends messages on its TCP destination without flow control, which
means that if the bandwidth of a TCP channel is less than the messages
received, it could intentionally drop messages, that's what is shown in
the DROPPED line.

2.0 supports flow control, which means that syslog-ng is trying to slow
its inputs down to the rate of its output pipes. This could slow down
applications at the end, but this is the only way to avoid dropping
messages.



-- 
Bazsi

_______________________________________________
syslog-ng maillist  -  syslog-ng at lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html