[syslog-ng] Log daemon for high volume of logs

[Federico Petronio]

In our case "high volume" is 83 pps (4 MB/hour) peaks probably are
higher but I don't have the number. As a test, we block with iptables
the incoming packets from the new host. After doing that, no other log
was miss, that's why we guess the problems is with the syslog buffer or
something related to it.

We also run the command "netstat -su" with the following result:

Udp:
    231803397 packets received
    6022 packets to unknown port received.
    123643084 packet receive errors
    117398380 packets sent

In this Debian Linux the output is different and not as detailed as the
one you showed, but it shows that 50% of the UDP packets has some kind
of problem. Do you know exactly what kind of problems generates "packet
receive errors";

Thank again,

K K wrote:
> On 1/29/07, Federico Petronio <petrus at activesec.biz> wrote:
>> I have a syslog (traditional) server that receives logs from a variety
>> of sources. Recently I added a new host to those that sends, but this
>> one generate high volume of logs. After adding this new one I notice
>> that many of the logs (form the new host and from the previous never
>> arrives to the syslog files). I did a little research and came to the
>> idea that syslog is not able to manage the large amount of logs.
> 
> Can you define 'high volume'?  How many packets per second, how many
> megabytes per second?
> 
> Remote syslog over UDP is inherently prone to lost packets.  No
> changes to the network or to the receiver will ever make UDP syslog
> 100% reliable.   Usually the first place that dropped packets occur is
> in the IP stack of the receiving host.  For example, here are
> statistics from a heavily loaded OpenBSD log server which has been up
> for 205 days:
> 
> $ netstat -s -p udp
> udp:
>        728902027 datagrams received
>        0 with incomplete header
>        0 with bad data length field
>        0 with bad checksum
>        92856 with no checksum
>        728809170 input packets hardware-checksummed
>        0 output packets hardware-checksummed
>        31254001 dropped due to no socket
>        0 broadcast/multicast datagrams dropped due to no socket
>        0 dropped due to missing IPsec protection
>        14461965 dropped due to full socket buffers
>        683186061 delivered
>        2492805 datagrams output
>        726705189 missed PCB cache
> 
> That shows 2% of UDP packets were "dropped due to full socket buffers",
> and 4% were "dropped due to no socket" (usually indicates that
> something is sending bogus packets to an inactive socket, bad or slow
> DNS responses, or packets lost because the syslogd was down and
> nothing was listening on UDP/514).
> 
> This is with an average of 41pps, and a peak rate somewhere north of
> 2000pps (3.54 Gigibytes/hour).  Most of the dropped packets occur
> during peak periods, and not all drops are fully accounted for by the
> operating system  or by the STATS events from syslog-ng (which are
> always 0 in my deployment).
> 
> One way to make remote syslog "reliable" is to switch to TCP as the
> transport protocol.  This usually requires replacing the syslog daemon
> on *both* the sender and on the receiver with a new syslogd (such as
> syslog-ng).  Sometimes this isn't possible, and you're stuck with
> doing the best you can with UDP (and living with as much as 7% packet
> loss).
> 
> 
>> I would like to know if syslog-ng is better than syslog in that subject.
> 
> Yes,
> 
>>   I know syslog-ng is better in other subjects, but currently I am
>> really urge to solve the high volume problem and that's why I interested
>> mainly in knowing about high volume support.
> 
> With fast hardware and a well-written syslog-ng.conf file, the daemon
> can handle a very high volume of log events, higher than what standard
> syslogd can handle on the same hardware and OS.
> 
> Kevin
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
> 

-- 
                                        Federico Petronio
                                        petrus at activesec.biz