[syslog-ng] Syslog-ng dropped packets.

[Vaibhav Goel]

The other thing I forgot to mention is that yes it is syslog-ng on the
receving end that is dropping packets.  We are logging a lot of web data
on a loghost.  We have no sending syslog-ng process.  We have our
application that connects directly to the syslog-ng port.  It is a java
application and uses log4j.  We wrote a custom "TCP appender" to connect
to port 514 and dump the log messages there.  Now it is possible I guess
that its dropping messages but unlikely.  We can do more investigation
in that.

-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu
[mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs
Scheidler
Sent: July 14, 2006 8:27 AM
To: Syslog-ng users' and developers' mailing list
Subject: RE: [syslog-ng] Syslog-ng dropped packets.

On Fri, 2006-07-14 at 06:21 -0700, Vaibhav Goel wrote:
> Hi Bazsi
> 
> We are still having issues with dropped packets under heavy load.  We
> increased the size of rmem_default and rmem_max drastically but we are
> not seeing much of an improvement....Maybe about 50% or so.
Currently,
> the sizes are 
> 
> $ > cat /proc/sys/net/core/rmem_default
> 786426
> $ > cat /proc/sys/net/core/rmem_max
> 1572852
> 
> Do you have any other suggestions?  We are running syslog-ng 1.6.11.
> 
> All the logging is done over TCP (except 1 UDP source but that doesn't
> generate a lot of logging data).

It would be important to know whether it is syslog-ng itself that drops
messages or something else in the system.

Do you have STATS lines in your logs? That shows how many lines were
dropped by syslog-ng itself. It is also possible that it is the sending
syslog-ng process that drops messages (in the case of TCP), you can
verify this by checking the STATS line of the sending syslog-ng process.
You can improve this situation somewhat by increasing log_fifo_size on
the sender.

1.6.x sends messages on its TCP destination without flow control, which
means that if the bandwidth of a TCP channel is less than the messages
received, it could intentionally drop messages, that's what is shown in
the DROPPED line.

2.0 supports flow control, which means that syslog-ng is trying to slow
its inputs down to the rate of its output pipes. This could slow down
applications at the end, but this is the only way to avoid dropping
messages.



-- 
Bazsi

_______________________________________________
syslog-ng maillist  -  syslog-ng at lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html