[syslog-ng] Insider 2023-12: compressed HTTP; packages; OpenObserve; duplicates;

[Peter Czanik (pczanik)]

Dear syslog-ng users,

This is the 115th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.

NEWS

Compressing HTTP traffic in syslog-ng
-------------------------------------
Network traffic is expensive in the cloud, and even a single syslog-ng instance can easily saturate the full bandwidth of a network connection. Compressing HTTP traffic was introduced in syslog-ng Version 4.4.0 and depending on your use case, you can cut down on your expenses on your networking or send more logs using the same budget or bandwidth.
Development of this feature was done using a locally installed OpenResty web server, and later tested using Sumologic. However, according to the docs it should also work with Splunk, Elasticsearch, and many other services accessible using the http() destination.
https://www.syslog-ng.com/community/b/blog/posts/compressing-http-traffic-in-syslog-ng

Why is a feature not available in the syslog-ng package?
--------------------------------------------------------
You can read about many interesting syslog-ng features in my blogs. However, it can happen that when you want to try them at home, you fail because the feature is missing. How can you solve such problems? In this blog, I discuss some of the possible solutions from installing sub-packages through using unofficial repositories, to upgrading your OS.
This blog focuses on RPM packages for openSUSE / SLES, Fedora / RHEL, and FreeBSD, because these are the packages I know – I am their maintainer. However, these problems and their solutions also apply to Debian / Ubuntu, and other Linux distributions.
https://www.syslog-ng.com/community/b/blog/posts/why-is-a-feature-not-available-in-the-syslog-ng-package

Sending logs to OpenObserve using syslog-ng
-------------------------------------------
A question was asked if syslog-ng can send logs to OpenObserve. It has an Elasticsearch compatible API for log ingestion, but syslog-ng is not mentioned in the documentation. My plan was to document how to modify the syslog-ng elasticsearch-http() destination, based on API documentation. However, as it turned out, OpenObserve has a ready to use syslog-ng configuration example in the web UI.
https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-openobserve-using-syslog-ng 

Removing duplicate messages with syslog-ng in a redundant logging environment
-----------------------------
Creating highly available servers is difficult. Sending logs to two (or more) servers and hoping that at least one of them can collect logs any time is a lot easier. Since network traffic and storage are cheap, redundancy is usually not a problem. However, once you also want to analyze your log messages using a SIEM or other software, you do not want duplicate log messages.
In this blog, I show you how you can make sure that each log message produced on your network reaches your SIEM system only once. Sending logs to a SIEM is expensive: both resource usage and licensing costs are much higher than at the log management level.
https://www.syslog-ng.com/community/b/blog/posts/removing-duplicate-messages-with-syslog-ng-in-a-redundant-logging-environment 

WEBINARS

* You can browse recordings of past webinars at https://www.syslog-ng.com/events/

Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/

Peter Czanik (CzP) <peter.czanik at oneidentity.com>
Balabit (a OneIdentity company) / syslog-ng upstream
https://syslog-ng.com/community/
https://twitter.com/PCzanik