Re: [syslog-ng] Compression on input and output ?
Hi Scot, I don't know about native compression abilities in syslog-ng. You could use TLS compression, but I don't know if that suits you. Other options would require that you write your compressing programs and include them in program source/destination or using unix sockets. Can you explain what is your use case, please? Maybe compression is not the goal, just the tool to achieve it and we can propose something else. Gabor On 13 Mar 2018 3:05 pm, "Scot" <scotrn@gmail.com> wrote: Hi, Has anyone looked at enabling compression between sources and destinations? Looks like Kafka is the only destination that supports compression by documentation. No inputs. Would be great to also compress between a syslog-ng relay and a syslog-ng target. Scot ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I think TLS compression is a good answer. I see options documented in logstash, ES, SPLUNK, SUMO. We have 2 syslog relays that send data over strait TCP right now to another syslog-ng master. I was looking for ways to optimize that communication as well as a network JSON input from logstash and how other connections to the above could be optimized. Just a thought without much thought put into it I guess. Scot On Tue, Mar 13, 2018 at 12:53 PM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hi Scot,
I don't know about native compression abilities in syslog-ng. You could use TLS compression, but I don't know if that suits you.
Other options would require that you write your compressing programs and include them in program source/destination or using unix sockets.
Can you explain what is your use case, please? Maybe compression is not the goal, just the tool to achieve it and we can propose something else.
Gabor
On 13 Mar 2018 3:05 pm, "Scot" <scotrn@gmail.com> wrote:
Hi,
Has anyone looked at enabling compression between sources and destinations?
Looks like Kafka is the only destination that supports compression by documentation. No inputs.
Would be great to also compress between a syslog-ng relay and a syslog-ng target.
Scot
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
"Scot" == Scot <scotrn@gmail.com> writes:
Scot> We have 2 syslog relays that send data over strait TCP right now to another Scot> syslog-ng master. Scot> I was looking for ways to optimize that communication as well as a network Scot> JSON input from logstash and how other connections to the above could be Scot> optimized. I think TLS compression is a viable, practical solution in this case. It's easy to set up, and as the compression applies to the whole stream, and isn't done on a per-message basis (which would be horribly inefficient), it has the potential of achieving compression ratios that offset the overhead of TLS. The alternatives (like using a custom program destination and source) are - in my opinion - considerably harder to set up well, because you'd need to figure out a way to get the compressed stuff from one host to another, and the tcp()/udp()/syslog() methods don't work here, because they aren't well suited for transfer of binary data, and generally operate on a per-message basis, while you'd rather send a continuous stream. -- |8]
I was thinking more of the Elastic, sumo and splunk way where the messages are buffered compressed and sent using some standard open source compression library. On Wed, Mar 14, 2018 at 7:41 AM, Gergely Nagy <algernon@balabit.com> wrote:
"Scot" == Scot <scotrn@gmail.com> writes:
Scot> We have 2 syslog relays that send data over strait TCP right now to another Scot> syslog-ng master. Scot> I was looking for ways to optimize that communication as well as a network Scot> JSON input from logstash and how other connections to the above could be Scot> optimized.
I think TLS compression is a viable, practical solution in this case. It's easy to set up, and as the compression applies to the whole stream, and isn't done on a per-message basis (which would be horribly inefficient), it has the potential of achieving compression ratios that offset the overhead of TLS.
The alternatives (like using a custom program destination and source) are - in my opinion - considerably harder to set up well, because you'd need to figure out a way to get the compressed stuff from one host to another, and the tcp()/udp()/syslog() methods don't work here, because they aren't well suited for transfer of binary data, and generally operate on a per-message basis, while you'd rather send a continuous stream.
-- |8] ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Gergely Nagy
-
Nagy, Gábor
-
Scot