The FAQ (http://www.campin.net/syslog-ng/faq.html) talks about syslog volume and mentions "sites that exceed five gigabytes per day of log data". Our log volume has been growing slowly over time, I recently checked my primary logger and noticed that the raw log volume for this past Monday (the 24-hour period from midnight to midnight) was 10,982,118,488 bytes, or 10.22GiB. Peak hourly volume was 913MiB. Peak one-second volume in that hour was 2,626 messages totaling 440KiB, no duplicates. System specs: OpenBSD on a Dell 2650 (single 2.8Ghz P4) running syslog-ng 1.6.X. Logs are written to a Dell PERC 3/Di SCSI RAID-0, as a 2-drive stripe. Kevin Kadow
Hello Kevin thx for this interesting quantitaive Info. Two additional Questions: what are your network specs : bandwith client -> loghost ? and are you seeing traces of dropped messages on the clients or the server ? TIA Micha
Peak hourly volume was 913MiB. Peak one-second volume in that hour was 2,626 messages totaling 440KiB, no duplicates. Kevin Kadow
On Sat, 12 Feb 2005 13:04:32 +0100, Michael Arndt <M.Arndt@science-computing.de> wrote:
thx for this interesting quantitaive Info.
On a related note, I believe that the reason syslog-ng handles this volume with little or no loss is related directly to how the daemon queues up packets and writes the data to disk in blocks instead of making a single write() call per message. I'll post the relevant sections of my syslog-ng.conf if there is interest.
Two additional Questions:
what are your network specs : bandwith client -> loghost ?
All of the sources are in the same physical facility, connected to the loghost via a dedicated 100/Full interface to a switch which only serves the loghost. Currently this is a 2924 switch, soon to be replaced with a 3524. Even at peak moments, the actual bandwidth seldom exceeds 5 megabits -- the real issue seems to be PPS.
and are you seeing traces of dropped messages on the clients or the server ?
I've done some primitive load testing with sending UDP syslog packets each containing a monotonically increasing sequence number, and the limiting factor when using UDP seems to be related to packets-per-second rather than bandwidth. From 'netstat -s', I see about 200,000 packets/year logged as "dropped due to full socket buffers". I've tuned the value of net.inet.udp.recvspace as high as it can safely be set. From the overall volume, this works out to about %0.001 loss. Kevin Kadow
Hi Kevin,
instead of making a single write() call per message. I'll post the relevant sections of my syslog-ng.conf if there is interest.
i am definitely interested ;-) Micha
On Mon, 14 Feb 2005 08:37:20 +0100, Michael Arndt wrote:
Kevin Kadow wrote:
Michael Arndt wrote: and are you seeing traces of dropped messages on the clients or the server ?
From 'netstat -s', I . . . "dropped due to full socket buffers".
I've set syslog-ng to log stats hourly, it always reports "dropped 0", the clients are configured to just toss out UDP messages directly to the MAC address of the log (static ARP entry on the clients), and so are unaware of success or failure. The deployment was designed a few years ago to handle anticipated log volume and with a key design criteria that logging cannot impact the performance of the log sources; even at the cost of dropped events. For example, if a PIX is configured to use TCP instead of UDP, the log stream becomes reliable, but the PIX will stop passing traffic if the logger fails (See http://tinyurl.com/4xcve). This may be considered a feature in certain deployments.
On a related note, I believe that the reason syslog-ng handles this volume with little or no loss is related directly to how the daemon queues up packets and writes the data to disk in blocks instead of making a single write() call per message. I'll post the relevant sections of my syslog-ng.conf if there is interest.
i am definitely interested ;-)
Here's the relevant lines from my "options" block: options { stats(3600); sync(256); log_fifo_size(4095); log_msg_size(2048); }; Depending on the block sized used by your filesystem and the stripe size used by your RAID (if you are using hardware RAID), you might want to set these values significantly higher. My log messages are generally quite short, so there is a big performance gain from buffering many messages per write. If you are trying to debug a problem by watching logs "in realtime" (e.g. via "tail -f") the buffering done by these syslog-ng options may become a liability, as the logger may buffer as many as 256 lines before writing to disk. Kevin Kadow
participants (2)
-
Kevin
-
Michael Arndt