[syslog-ng] Log daemon for high volume of logs

[K K]

On 1/29/07, Federico Petronio <petrus at activesec.biz> wrote:
> I have a syslog (traditional) server that receives logs from a variety
> of sources. Recently I added a new host to those that sends, but this
> one generate high volume of logs. After adding this new one I notice
> that many of the logs (form the new host and from the previous never
> arrives to the syslog files). I did a little research and came to the
> idea that syslog is not able to manage the large amount of logs.

Can you define 'high volume'?  How many packets per second, how many
megabytes per second?

Remote syslog over UDP is inherently prone to lost packets.  No
changes to the network or to the receiver will ever make UDP syslog
100% reliable.   Usually the first place that dropped packets occur is
in the IP stack of the receiving host.  For example, here are
statistics from a heavily loaded OpenBSD log server which has been up
for 205 days:

$ netstat -s -p udp
udp:
        728902027 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        92856 with no checksum
        728809170 input packets hardware-checksummed
        0 output packets hardware-checksummed
        31254001 dropped due to no socket
        0 broadcast/multicast datagrams dropped due to no socket
        0 dropped due to missing IPsec protection
        14461965 dropped due to full socket buffers
        683186061 delivered
        2492805 datagrams output
        726705189 missed PCB cache

That shows 2% of UDP packets were "dropped due to full socket buffers",
and 4% were "dropped due to no socket" (usually indicates that
something is sending bogus packets to an inactive socket, bad or slow
DNS responses, or packets lost because the syslogd was down and
nothing was listening on UDP/514).

This is with an average of 41pps, and a peak rate somewhere north of
2000pps (3.54 Gigibytes/hour).  Most of the dropped packets occur
during peak periods, and not all drops are fully accounted for by the
operating system  or by the STATS events from syslog-ng (which are
always 0 in my deployment).

One way to make remote syslog "reliable" is to switch to TCP as the
transport protocol.  This usually requires replacing the syslog daemon
on *both* the sender and on the receiver with a new syslogd (such as
syslog-ng).  Sometimes this isn't possible, and you're stuck with
doing the best you can with UDP (and living with as much as 7% packet
loss).


> I would like to know if syslog-ng is better than syslog in that subject.

Yes,

>   I know syslog-ng is better in other subjects, but currently I am
> really urge to solve the high volume problem and that's why I interested
> mainly in knowing about high volume support.

With fast hardware and a well-written syslog-ng.conf file, the daemon
can handle a very high volume of log events, higher than what standard
syslogd can handle on the same hardware and OS.

Kevin