On Thu, 2007-05-24 at 22:37 +0530, Raghu (Lists) wrote:
Hi all,
I am working on a project to build distributed syslog-ing system for a very lager enterprise with offices all across the globe. Below are the main objectives:
1. Support for primarily network devices, like ciscos, netscreens, junipers etc 2. Minimum or no loss of messages when network fails 3. Central storage of all syslog messages
Could you please give me your ideas or point me to any documentation that deals with such designs?
First of all, please note that I'm working for BalaBit, so I'm biased. The Open Source Edition of syslog-ng falls short on your second objective: it will start losing messages once its memory based queues are full. However syslog-ng can be a solution, but you need the Premium Edition of syslog-ng. The Premium Edition of syslog-ng is capable of using disk-space as a message queue, so you can get through outages by allocating a large enough chunk of disk-space for syslog-ng to spool messages. The configuration would look like something like this: destination d_central { tcp("1.2.3.4" log_disk_fifo_size(1073741824)); }; This allocates about 1GiB space for destination spooling. [1] If you don't have this amount of space in syslog sources or you are using appliances where installing syslog-ng is not an option, you can use syslog relays. E.g. your infrastructure would be something like this: source1 \ source2 \ source3 |-- relay -- center ... / sourceN / Relays need to be deployed close to the source systems (e.g. on the same physical network, e.g. the probability of loss is low), then relay would be talking to the center using a possibly TLS encrypted TCP connection. You need to size the storage in your relays according to your requirements. Of course you can add another layer of relays (e.g. one dumber device on the same network as the sources, and a larger device somewhere in the middle, collecting several relays' worth of traffic). If you can deploy syslog-ng on the source systems, you could also enable flow-control. [2] [1]Using disk-buffering: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07s04.html [2]Flow control: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07s03.html -- Bazsi