[syslog-ng]Update for syslog-ng FAQ

[Kevin]

On Mon, 14 Feb 2005 08:37:20 +0100, Michael Arndt wrote:
>>  Kevin Kadow wrote:
>>> Michael Arndt wrote:
>>> and are you seeing traces of dropped messages on
>>>  the clients or the server ?
>>
>> From 'netstat -s', I . . .  "dropped due to full socket buffers".

I've set syslog-ng to log stats hourly, it always reports "dropped 0",
the clients are configured to just toss out UDP messages directly
to the MAC address of the log (static ARP entry on the clients),
and so are unaware of success or failure.

The deployment was designed a few years ago to handle anticipated
log volume and with a key design criteria that logging cannot impact
the performance of the log sources; even at the cost of dropped events.

For example, if a PIX is configured to use TCP instead of UDP, the log
stream becomes reliable, but the PIX will stop passing traffic if the
logger fails (See http://tinyurl.com/4xcve).  This may be considered a
feature in certain deployments.


>> On a related note, I believe that the reason syslog-ng handles
>> this volume with little or no loss is related directly to how the
>> daemon queues up packets and writes the data to disk in blocks
>> instead of making a single write() call per message.  I'll post the
>> relevant sections of my syslog-ng.conf if there is interest.
> 
> i am definitely interested ;-)

Here's the relevant lines from my "options" block:

   options {
     stats(3600);
     sync(256);
     log_fifo_size(4095);
     log_msg_size(2048);
   };

Depending on the block sized used by your filesystem and the 
stripe size used by your RAID (if you are using hardware RAID), 
you might want to set these values significantly higher.

My log messages are generally quite short, so there is a big
performance gain from buffering many messages per write.

If you are trying to debug a problem by watching logs "in realtime"
(e.g. via "tail -f") the buffering done by these syslog-ng options may
become a liability, as the logger may buffer as many as 256 lines
before writing to disk.

Kevin Kadow