Dear list members, I have a syslog (traditional) server that receives logs from a variety of sources. Recently I added a new host to those that sends, but this one generate high volume of logs. After adding this new one I notice that many of the logs (form the new host and from the previous never arrives to the syslog files). I did a little research and came to the idea that syslog is not able to manage the large amount of logs. I would like to know if syslog-ng is better than syslog in that subject. I know syslog-ng is better in other subjects, but currently I am really urge to solve the high volume problem and that's why I interested mainly in knowing about high volume support. Thank you very much. Best regards, -- Federico Petronio petrus@activesec.biz
On 1/29/07, Federico Petronio <petrus@activesec.biz> wrote:
I have a syslog (traditional) server that receives logs from a variety of sources. Recently I added a new host to those that sends, but this one generate high volume of logs. After adding this new one I notice that many of the logs (form the new host and from the previous never arrives to the syslog files). I did a little research and came to the idea that syslog is not able to manage the large amount of logs.
Can you define 'high volume'? How many packets per second, how many megabytes per second? Remote syslog over UDP is inherently prone to lost packets. No changes to the network or to the receiver will ever make UDP syslog 100% reliable. Usually the first place that dropped packets occur is in the IP stack of the receiving host. For example, here are statistics from a heavily loaded OpenBSD log server which has been up for 205 days: $ netstat -s -p udp udp: 728902027 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 92856 with no checksum 728809170 input packets hardware-checksummed 0 output packets hardware-checksummed 31254001 dropped due to no socket 0 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 14461965 dropped due to full socket buffers 683186061 delivered 2492805 datagrams output 726705189 missed PCB cache That shows 2% of UDP packets were "dropped due to full socket buffers", and 4% were "dropped due to no socket" (usually indicates that something is sending bogus packets to an inactive socket, bad or slow DNS responses, or packets lost because the syslogd was down and nothing was listening on UDP/514). This is with an average of 41pps, and a peak rate somewhere north of 2000pps (3.54 Gigibytes/hour). Most of the dropped packets occur during peak periods, and not all drops are fully accounted for by the operating system or by the STATS events from syslog-ng (which are always 0 in my deployment). One way to make remote syslog "reliable" is to switch to TCP as the transport protocol. This usually requires replacing the syslog daemon on *both* the sender and on the receiver with a new syslogd (such as syslog-ng). Sometimes this isn't possible, and you're stuck with doing the best you can with UDP (and living with as much as 7% packet loss).
I would like to know if syslog-ng is better than syslog in that subject.
Yes,
I know syslog-ng is better in other subjects, but currently I am really urge to solve the high volume problem and that's why I interested mainly in knowing about high volume support.
With fast hardware and a well-written syslog-ng.conf file, the daemon can handle a very high volume of log events, higher than what standard syslogd can handle on the same hardware and OS. Kevin
In our case "high volume" is 83 pps (4 MB/hour) peaks probably are higher but I don't have the number. As a test, we block with iptables the incoming packets from the new host. After doing that, no other log was miss, that's why we guess the problems is with the syslog buffer or something related to it. We also run the command "netstat -su" with the following result: Udp: 231803397 packets received 6022 packets to unknown port received. 123643084 packet receive errors 117398380 packets sent In this Debian Linux the output is different and not as detailed as the one you showed, but it shows that 50% of the UDP packets has some kind of problem. Do you know exactly what kind of problems generates "packet receive errors"; Thank again, K K wrote:
On 1/29/07, Federico Petronio <petrus@activesec.biz> wrote:
I have a syslog (traditional) server that receives logs from a variety of sources. Recently I added a new host to those that sends, but this one generate high volume of logs. After adding this new one I notice that many of the logs (form the new host and from the previous never arrives to the syslog files). I did a little research and came to the idea that syslog is not able to manage the large amount of logs.
Can you define 'high volume'? How many packets per second, how many megabytes per second?
Remote syslog over UDP is inherently prone to lost packets. No changes to the network or to the receiver will ever make UDP syslog 100% reliable. Usually the first place that dropped packets occur is in the IP stack of the receiving host. For example, here are statistics from a heavily loaded OpenBSD log server which has been up for 205 days:
$ netstat -s -p udp udp: 728902027 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 92856 with no checksum 728809170 input packets hardware-checksummed 0 output packets hardware-checksummed 31254001 dropped due to no socket 0 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 14461965 dropped due to full socket buffers 683186061 delivered 2492805 datagrams output 726705189 missed PCB cache
That shows 2% of UDP packets were "dropped due to full socket buffers", and 4% were "dropped due to no socket" (usually indicates that something is sending bogus packets to an inactive socket, bad or slow DNS responses, or packets lost because the syslogd was down and nothing was listening on UDP/514).
This is with an average of 41pps, and a peak rate somewhere north of 2000pps (3.54 Gigibytes/hour). Most of the dropped packets occur during peak periods, and not all drops are fully accounted for by the operating system or by the STATS events from syslog-ng (which are always 0 in my deployment).
One way to make remote syslog "reliable" is to switch to TCP as the transport protocol. This usually requires replacing the syslog daemon on *both* the sender and on the receiver with a new syslogd (such as syslog-ng). Sometimes this isn't possible, and you're stuck with doing the best you can with UDP (and living with as much as 7% packet loss).
I would like to know if syslog-ng is better than syslog in that subject.
Yes,
I know syslog-ng is better in other subjects, but currently I am really urge to solve the high volume problem and that's why I interested mainly in knowing about high volume support.
With fast hardware and a well-written syslog-ng.conf file, the daemon can handle a very high volume of log events, higher than what standard syslogd can handle on the same hardware and OS.
Kevin _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Federico Petronio petrus@activesec.biz
On Mon, 2007-01-29 at 17:54 -0300, Federico Petronio wrote:
In our case "high volume" is 83 pps (4 MB/hour) peaks probably are higher but I don't have the number. As a test, we block with iptables the incoming packets from the new host. After doing that, no other log was miss, that's why we guess the problems is with the syslog buffer or something related to it.
We also run the command "netstat -su" with the following result:
Udp: 231803397 packets received 6022 packets to unknown port received. 123643084 packet receive errors 117398380 packets sent
In this Debian Linux the output is different and not as detailed as the one you showed, but it shows that 50% of the UDP packets has some kind of problem. Do you know exactly what kind of problems generates "packet receive errors";
try to increase the socket receive buffer. You can do that with so_rcvbuf() option in syslog-ng, but you can tweak kernel tunables as well. -- Bazsi
On Tue, Jan 30, 2007 at 06:34:09PM +0100, Balazs Scheidler wrote:
try to increase the socket receive buffer. You can do that with so_rcvbuf() option in syslog-ng, but you can tweak kernel tunables as well.
At my current job, I was never able to handle my syslog spikes with UDP, even after increasing the UDP receive buffer. My troubles didn't go away until I deployed syslog-ng everywhere and used TCP for all syslog traffic (from UNIX hosts anyways). I only mention this because I want to make it crystal clear that you'll probably see the same problems after deploying syslog-ng on your central log server(s). The problem is lower in the stack than that. If this was already clear to you, then sorry for the wasted bandwidth. :) HTH, -- Nate "Do the right thing. It will gratify some people and astonish the rest." - Samuel Clemens
participants (4)
-
Balazs Scheidler
-
Federico Petronio
-
K K
-
Nate Campi