[syslog-ng] Compressing syslog traffic across a WAN?

[K K]

On 4/9/07, Federico Ceratto <federico.ceratto at gmail.com> wrote:
> K K wrote:
> > Has anybody found a good way to take a high-volume stream of UDP
> > syslog packets, aggregate and compress the packets, and then recreate
> > them, with the original source IPs, at the other end of a WAN circuit?
> The need of preserving the original source IP adresses makes the problem
> tricky.
> I guess your logs are important ;) so crafting up some scripts wouldn't
> be so appealing. Why don't letting your two syslog-ng instances
> communicate over a VPN tunnel? OpenVPN is very mature and robust, it
> supports traffic compression and encryption. Also you can make the
> tunnel lossless using TCP (if syslog is using UDP)

I've considered a couple of different VPN approaches, but so far all
of the approaches tend to have poor compression, and even in TCP mode
do not combine multiple compressed UDP packets into a single packet,
so there is no reduction in packets-per-second, just in bps.

I'm tempted to just write out EIQ-compatible text log files on the
originating syslog server and just bulk transfer these logs over to
the EIQ "appliance" every X minutes, though I doubt the vendor will
support this.

Kevin