[syslog-ng] Compressing syslog traffic across a WAN?

[Alexander Clouter]

Hi,

K K <kkadow at gmail.com> [20070322 14:02:27 -0500]:
>
> Has anybody found a good way to take a high-volume stream of UDP
> syslog packets, aggregate and compress the packets, and then recreate
> them, with the original source IPs, at the other end of a WAN circuit?
> 
> Looks like Riverbed cannot optimize UDP syslog?.  Encapsulating the
> packets in an IPSEC tunnel with compression would make the packets
> smaller, but not reduce the packets-per-second.
> 
This would be a 'trivial' thing to do with a Perl script....it would not even 
have to decode the packets, just send and relay the messages at the other 
end.

A short term hack would be to use ppp (load on the compression) with netcat 
over TCP.

A quick search over at freshmeat.net (should always be a first port of call 
for people) is:

http://www.winton.org.uk/zebedee/manual.html

Cheers

Alex

> Tolstoy Version:
> I have a number of 'appliance' hosts on the west coast which generate
> high volumes of syslog events, which I need to forward to an
> 'appliance' log analysis server (EIQ) in the midwest.  The version of
> EIQ we're stuck with only supports UDP, and uses the source IP of the
> packet to decide which host the event occurred on -- it cannot take
> the embedded hostname in the packet and use that.
> 
> What we're doing right now is configuring all the west coast
> appliances send their syslog events to a syslog-ng server locally on
> the same network, which then filters out the junk events and uses
> spoof-source to forward these UDP packets across the slow WAN circuit
> to the EIQ appliances in the Midwest:
> 
> Source \
> Source  --UDP-- syslog-ng --UDP-over-WAN--  EIQ
> Source /
> 
> I could consider sending these events via TCP syslog to a syslog-ng
> server in the Midwest, and that would then send a copy of the packets
> via UDP to EIQ, but in this approach, I gather that we wouldn't be
> able to use spoof-source to regenerate the UDP packet with the
> original source IP?
> 
> 
> Thanks,
> 
> Kevin
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
>