[syslog-ng] Architecture redundancy

[Andrew Fort]

John.Dickinson at nominet.org.uk wrote:
> I have been doing this and recent messages on this list suggest many 
> others are too. The solution that I settled on was to cluster the two 
> syslog-ng servers so that syslog-ng only ran on one server and failed over 
> to the other if the first server died. The logs are written to a clustered 
> filesystem on a SAN. I did consider running syslog-ng on both servers and 
> then trying to do some kind of log file reconciliation every night but 
> decided that would be too hard. To do the clustering we are using Poyserve 
> but there are lots of other clustering systems out there that you could 
> use.
>  
> The big problem that I have is the loss of log messages during failover. 
> If anyone else has suggested configs or ideas on minimising this I would 
> be very interested to hear.
> 
> Thanks
> John

I'm using two syslog-ng listeners, listening to the same anycast address 
(it's a loopback address on the hosts), and a single database written to 
by both hosts.  I'm using static routing but you can improve failover 
times using a bgpd to announce the loopback via BGP.

On UDP traffic I get usually one packet drop only, thus usually just one 
message dropped.  For TCP logs it takes longer, since the TCP sequence 
numbers don't match for the 'backup' server that has taken over the 
traffic, and the session stales out on the sender (receiver ignores the 
sender).  So, on all syslog-ng hosts I use these options:

options {
         time_reopen(5);
};

Which will cause the TCP session to be killed and re-established in 5 
seconds if it goes stale (default is 60 seconds).

Fortunately it's rarely required in our present architecture if you have 
just two servers, since one is effectively a backup (though hosts 
connected to the same router as the 'backup' syslog-ng host will log to it).

Anyone have a good solution for the "same filename, multiple data files" 
data consolidation approach?  I used a quick perl hack to combine and 
sort, but am wondering if someone has a better solution for this problem?

-andrew