On 4/9/07, Federico Ceratto <federico.ceratto@gmail.com> wrote:
K K wrote:
Has anybody found a good way to take a high-volume stream of UDP syslog packets, aggregate and compress the packets, and then recreate them, with the original source IPs, at the other end of a WAN circuit? The need of preserving the original source IP adresses makes the problem tricky. I guess your logs are important ;) so crafting up some scripts wouldn't be so appealing. Why don't letting your two syslog-ng instances communicate over a VPN tunnel? OpenVPN is very mature and robust, it supports traffic compression and encryption. Also you can make the tunnel lossless using TCP (if syslog is using UDP)
I've considered a couple of different VPN approaches, but so far all of the approaches tend to have poor compression, and even in TCP mode do not combine multiple compressed UDP packets into a single packet, so there is no reduction in packets-per-second, just in bps. I'm tempted to just write out EIQ-compatible text log files on the originating syslog server and just bulk transfer these logs over to the EIQ "appliance" every X minutes, though I doubt the vendor will support this. Kevin