Compressing syslog traffic across a WAN?
Has anybody found a good way to take a high-volume stream of UDP syslog packets, aggregate and compress the packets, and then recreate them, with the original source IPs, at the other end of a WAN circuit? Looks like Riverbed cannot optimize UDP syslog?. Encapsulating the packets in an IPSEC tunnel with compression would make the packets smaller, but not reduce the packets-per-second. Tolstoy Version: I have a number of 'appliance' hosts on the west coast which generate high volumes of syslog events, which I need to forward to an 'appliance' log analysis server (EIQ) in the midwest. The version of EIQ we're stuck with only supports UDP, and uses the source IP of the packet to decide which host the event occurred on -- it cannot take the embedded hostname in the packet and use that. What we're doing right now is configuring all the west coast appliances send their syslog events to a syslog-ng server locally on the same network, which then filters out the junk events and uses spoof-source to forward these UDP packets across the slow WAN circuit to the EIQ appliances in the Midwest: Source \ Source --UDP-- syslog-ng --UDP-over-WAN-- EIQ Source / I could consider sending these events via TCP syslog to a syslog-ng server in the Midwest, and that would then send a copy of the packets via UDP to EIQ, but in this approach, I gather that we wouldn't be able to use spoof-source to regenerate the UDP packet with the original source IP? Thanks, Kevin
Hi, K K <kkadow@gmail.com> [20070322 14:02:27 -0500]:
Has anybody found a good way to take a high-volume stream of UDP syslog packets, aggregate and compress the packets, and then recreate them, with the original source IPs, at the other end of a WAN circuit?
Looks like Riverbed cannot optimize UDP syslog?. Encapsulating the packets in an IPSEC tunnel with compression would make the packets smaller, but not reduce the packets-per-second.
This would be a 'trivial' thing to do with a Perl script....it would not even have to decode the packets, just send and relay the messages at the other end. A short term hack would be to use ppp (load on the compression) with netcat over TCP. A quick search over at freshmeat.net (should always be a first port of call for people) is: http://www.winton.org.uk/zebedee/manual.html Cheers Alex
Tolstoy Version: I have a number of 'appliance' hosts on the west coast which generate high volumes of syslog events, which I need to forward to an 'appliance' log analysis server (EIQ) in the midwest. The version of EIQ we're stuck with only supports UDP, and uses the source IP of the packet to decide which host the event occurred on -- it cannot take the embedded hostname in the packet and use that.
What we're doing right now is configuring all the west coast appliances send their syslog events to a syslog-ng server locally on the same network, which then filters out the junk events and uses spoof-source to forward these UDP packets across the slow WAN circuit to the EIQ appliances in the Midwest:
Source \ Source --UDP-- syslog-ng --UDP-over-WAN-- EIQ Source /
I could consider sending these events via TCP syslog to a syslog-ng server in the Midwest, and that would then send a copy of the packets via UDP to EIQ, but in this approach, I gather that we wouldn't be able to use spoof-source to regenerate the UDP packet with the original source IP?
Thanks,
Kevin _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
K K wrote:
Has anybody found a good way to take a high-volume stream of UDP syslog packets, aggregate and compress the packets, and then recreate them, with the original source IPs, at the other end of a WAN circuit? The need of preserving the original source IP adresses makes the problem tricky. I guess your logs are important ;) so crafting up some scripts wouldn't be so appealing. Why don't letting your two syslog-ng instances communicate over a VPN tunnel? OpenVPN is very mature and robust, it supports traffic compression and encryption. Also you can make the tunnel lossless using TCP (if syslog is using UDP)
Just my 2 cents -- Federico Ceratto - Linux System Administrator Every man takes the limits of his own field of vision for the limits of the world. - A. Schopenhauer
On 4/9/07, Federico Ceratto <federico.ceratto@gmail.com> wrote:
K K wrote:
Has anybody found a good way to take a high-volume stream of UDP syslog packets, aggregate and compress the packets, and then recreate them, with the original source IPs, at the other end of a WAN circuit? The need of preserving the original source IP adresses makes the problem tricky. I guess your logs are important ;) so crafting up some scripts wouldn't be so appealing. Why don't letting your two syslog-ng instances communicate over a VPN tunnel? OpenVPN is very mature and robust, it supports traffic compression and encryption. Also you can make the tunnel lossless using TCP (if syslog is using UDP)
I've considered a couple of different VPN approaches, but so far all of the approaches tend to have poor compression, and even in TCP mode do not combine multiple compressed UDP packets into a single packet, so there is no reduction in packets-per-second, just in bps. I'm tempted to just write out EIQ-compatible text log files on the originating syslog server and just bulk transfer these logs over to the EIQ "appliance" every X minutes, though I doubt the vendor will support this. Kevin
participants (3)
-
Alexander Clouter
-
Federico Ceratto
-
K K