Yup... Below are some of the stats. As you can see, in peak hours, the rate of the dropped messages is huge. I have increased the log_fifo_size in the global options to 4096 (from 2000) options { sync (0); time_reopen (10); log_fifo_size (4096); long_hostnames (on); use_dns (yes); } Jul 13 15:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 9445 Jul 13 15:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 18944 Jul 13 15:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 10517 Jul 13 15:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 12071 Jul 13 16:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 17505 Jul 13 16:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 12226 Jul 13 16:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 8276 Jul 13 16:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 13717 Jul 13 16:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 14688 Jul 13 16:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 4136 Jul 13 17:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 5506 Jul 13 17:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2328 Jul 13 17:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1754 Jul 13 17:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 3253 Jul 13 17:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2449 Jul 13 17:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1023 Jul 13 18:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1868 Jul 13 18:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1541 Jul 13 18:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1863 Jul 13 18:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 380 Jul 13 18:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1340 Jul 13 18:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1241 Jul 13 19:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1693 Jul 13 19:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1858 Jul 13 19:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 19:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 651 Jul 13 19:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 739 Jul 13 19:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1894 Jul 13 20:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 294 Jul 13 20:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 20:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 586 Jul 13 20:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 996 Jul 13 20:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 203 Jul 13 20:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 594 Jul 13 21:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2365 Jul 13 21:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1158 Jul 13 21:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1065 Jul 13 21:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 949 Jul 13 21:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2341 Jul 13 21:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 98 Jul 13 22:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 382 Jul 13 22:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 50 Jul 13 22:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 84 Jul 13 22:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 293 Jul 13 22:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 54 Jul 13 22:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 351 Jul 13 23:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 6 Jul 13 23:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 493 Jul 13 23:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 23:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 23:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 23:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 623 Jul 14 07:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 07:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 07:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 10755 Jul 14 07:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 6718 Jul 14 07:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 17484 Jul 14 07:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 30382 Jul 14 08:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 32052 Jul 14 08:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 47914 Jul 14 08:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 44719 Jul 14 08:41:31 s_syslogng@clxp01ml syslog-ng[8094]: STATS: dropped 37327 Jul 14 08:51:31 s_syslogng@clxp01ml syslog-ng[8094]: STATS: dropped 49299 -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: July 14, 2006 8:27 AM To: Syslog-ng users' and developers' mailing list Subject: RE: [syslog-ng] Syslog-ng dropped packets. On Fri, 2006-07-14 at 06:21 -0700, Vaibhav Goel wrote:
Hi Bazsi
We are still having issues with dropped packets under heavy load. We increased the size of rmem_default and rmem_max drastically but we are not seeing much of an improvement....Maybe about 50% or so. Currently, the sizes are
$ > cat /proc/sys/net/core/rmem_default 786426 $ > cat /proc/sys/net/core/rmem_max 1572852
Do you have any other suggestions? We are running syslog-ng 1.6.11.
All the logging is done over TCP (except 1 UDP source but that doesn't generate a lot of logging data).
It would be important to know whether it is syslog-ng itself that drops messages or something else in the system. Do you have STATS lines in your logs? That shows how many lines were dropped by syslog-ng itself. It is also possible that it is the sending syslog-ng process that drops messages (in the case of TCP), you can verify this by checking the STATS line of the sending syslog-ng process. You can improve this situation somewhat by increasing log_fifo_size on the sender. 1.6.x sends messages on its TCP destination without flow control, which means that if the bandwidth of a TCP channel is less than the messages received, it could intentionally drop messages, that's what is shown in the DROPPED line. 2.0 supports flow control, which means that syslog-ng is trying to slow its inputs down to the rate of its output pipes. This could slow down applications at the end, but this is the only way to avoid dropping messages. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html