[syslog-ng] STATS: high amount of dropped messages

[Balazs Scheidler]

On Wed, 2008-04-09 at 17:12 -0400, Jeffrey Chandler wrote:
> Hello,
> 
> I am using syslog-ng 1.6.12 on Solaris 10 (sparc).  I am seeing a
> large amount of dropped message every time 
> 
> Apr  9 20:14:18 syslog syslog-ng[16754]: STATS: dropped 1068470
> Apr  9 20:24:18 syslog syslog-ng[16754]: STATS: dropped 1031716
> Apr  9 20:34:18 syslog syslog-ng[16754]: STATS: dropped 997105
> Apr  9 20:44:18 syslog syslog-ng[16754]: STATS: dropped 943151
> Apr  9 20:54:18 syslog syslog-ng[16754]: STATS: dropped 1016377
> Apr  9 21:04:18 syslog syslog-ng[16754]: STATS: dropped 992911
> 
> In previous posts, I have seen that this does not relate to log
> entries written to disk, but rather to a different destination, ie a
> pipe.  In the case of our environment, we pipe all of the data which
> comes into syslog-ng to a pipe which is then forwarded to ArcSight
> through an agent running on the system.  With the number of entries
> showing up as "dropped," would you recommend a different method to
> achieve this same task?  Is this also a 1 to 1 correlation of messages
> that are not sent to the pipe?

pipe should be ok from a transport perspective, the problem is probably
caused by the fact that arcsight is too slow processing messages.

There are two solutions:
 1) if there are only peaks that arcsight is unable to process but there
are otherwise idle periods, then you could increase the FIFO size in
syslog-ng, this will level the load on arcsight, although with some
possible latency instead of drops

 2) upgrade to 2.0 and enable flow-control and ensure that all hops
between the sending application and arcsight is flow-controlled too.
This will prevent message loss at the price of slowing applications down
to the processing rate of arcsight (of course buffer sizes help again,
to prevent application slow-down from actually happening)

 3) upgrade to 2.1 PE and use disk buffering which is able to level
peaks even more with the use of the disk as spooling area

-- 
Bazsi