Insider 2018-12: 3.19 release; optimizing Splunk; Python source; HTTP batch;
Dear syslog-ng users, This is the 71st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.19 of syslog-ng released ---------------------------------- Version 3.19 of syslog-ng has been released with plenty of new features and bugfixes. Performance of the HTTP destination improved thanks to load-balancing to multiple servers. You can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. The new Slack destination allows you to send alerts to a Slack channel. Read the complete list of changes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1. Optimize your Splunk infrastructure using new syslog-ng features ------------------------------------------------------- Learn how to use less resources for better performance in Splunk! Many people have been using syslog-ng for decades without knowing that it receives new features as well as bugfixes. While many Linux utilities are practically in maintenance mode, syslog-ng keeps evolving constantly. A strong focus in recent years has been on message parsing and destination drivers. After my talk at Suricon, Splunk users explained how they will change their syslog-ng configurations to optimize their Splunk infrastructure. https://www.syslog-ng.com/community/b/blog/posts/optimize-your-splunk-infras... Python source in syslog-ng -------------------------- Using syslog-ng 3.18 and newer releases, you can write new source drivers for syslog-ng in Python. While performance is not as good as C, you gain flexibility and ease of implementation. There are quite a few log sources without a ready to use C API, but with a Python API. Using the Python source of syslog-ng you can leverage these. https://www.syslog-ng.com/community/b/blog/posts/python-source-in-syslog-ng Bulk mode message sending to Elasticsearch with syslog-ng http() destination --------------------------------------------------------- Learn how to send log messages in bulk mode to your Elasticsearch server with syslog-ng. Bulk mode offers better performance, because it sends multiple log messages in a single POST request. https://www.syslog-ng.com/community/b/blog/posts/bulk-mode-message-sending-t... WEBINARS You can watch our past webinars: * Log ingestion to Splunk HEC: https://www.brighttalk.com/webcast/16207/338190 * High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik
Hi, Is there any real deep-dive document showing why it is actually better to send from the relay to HEC destination than send simply to a syslog destination? I was running a pilot project where we pumped firewall logs from syslog-ng PE 7 relay to Splunk Enterprise on plain syslog/tcp (of course with disc buffer) and I never experienced any issue in this scenario, so I've tried to find a detailed document on performance wise differences between syslog/tcp and HEC destinations Do you have such a whitepaper? Thx L: On Thu, Dec 13, 2018 at 4:00 PM Czanik, Péter <peter.czanik@balabit.com> wrote:
Dear syslog-ng users,
This is the 71st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.
NEWS
Version 3.19 of syslog-ng released ----------------------------------
Version 3.19 of syslog-ng has been released with plenty of new features and bugfixes. Performance of the HTTP destination improved thanks to load-balancing to multiple servers. You can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. The new Slack destination allows you to send alerts to a Slack channel.
Read the complete list of changes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1.
Optimize your Splunk infrastructure using new syslog-ng features -------------------------------------------------------
Learn how to use less resources for better performance in Splunk! Many people have been using syslog-ng for decades without knowing that it receives new features as well as bugfixes. While many Linux utilities are practically in maintenance mode, syslog-ng keeps evolving constantly. A strong focus in recent years has been on message parsing and destination drivers.
After my talk at Suricon, Splunk users explained how they will change their syslog-ng configurations to optimize their Splunk infrastructure.
https://www.syslog-ng.com/community/b/blog/posts/optimize-your-splunk-infras...
Python source in syslog-ng --------------------------
Using syslog-ng 3.18 and newer releases, you can write new source drivers for syslog-ng in Python. While performance is not as good as C, you gain flexibility and ease of implementation. There are quite a few log sources without a ready to use C API, but with a Python API. Using the Python source of syslog-ng you can leverage these.
https://www.syslog-ng.com/community/b/blog/posts/python-source-in-syslog-ng
Bulk mode message sending to Elasticsearch with syslog-ng http() destination ---------------------------------------------------------
Learn how to send log messages in bulk mode to your Elasticsearch server with syslog-ng. Bulk mode offers better performance, because it sends multiple log messages in a single POST request.
https://www.syslog-ng.com/community/b/blog/posts/bulk-mode-message-sending-t...
WEBINARS
You can watch our past webinars:
* Log ingestion to Splunk HEC: https://www.brighttalk.com/webcast/16207/338190
* High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943
Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
There was a talk in 2017 .conf where there's an entire slide dedicated to that. If you look at the video recording you'll see why Splunk engineers don't recommend doing that. https://conf.splunk.com/conf-online.html?search=to%20hec%20with%20syslog#/ It's mostly about scalability. On Thu, Dec 13, 2018 at 4:25 PM Pal, Laszlo <vlad@vlad.hu> wrote:
Hi,
Is there any real deep-dive document showing why it is actually better to send from the relay to HEC destination than send simply to a syslog destination? I was running a pilot project where we pumped firewall logs from syslog-ng PE 7 relay to Splunk Enterprise on plain syslog/tcp (of course with disc buffer) and I never experienced any issue in this scenario, so I've tried to find a detailed document on performance wise differences between syslog/tcp and HEC destinations
Do you have such a whitepaper?
Thx L:
On Thu, Dec 13, 2018 at 4:00 PM Czanik, Péter <peter.czanik@balabit.com> wrote:
Dear syslog-ng users,
This is the 71st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.
NEWS
Version 3.19 of syslog-ng released ----------------------------------
Version 3.19 of syslog-ng has been released with plenty of new features and bugfixes. Performance of the HTTP destination improved thanks to load-balancing to multiple servers. You can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. The new Slack destination allows you to send alerts to a Slack channel.
Read the complete list of changes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1.
Optimize your Splunk infrastructure using new syslog-ng features -------------------------------------------------------
Learn how to use less resources for better performance in Splunk! Many people have been using syslog-ng for decades without knowing that it receives new features as well as bugfixes. While many Linux utilities are practically in maintenance mode, syslog-ng keeps evolving constantly. A strong focus in recent years has been on message parsing and destination drivers.
After my talk at Suricon, Splunk users explained how they will change their syslog-ng configurations to optimize their Splunk infrastructure.
https://www.syslog-ng.com/community/b/blog/posts/optimize-your-splunk-infras...
Python source in syslog-ng --------------------------
Using syslog-ng 3.18 and newer releases, you can write new source drivers for syslog-ng in Python. While performance is not as good as C, you gain flexibility and ease of implementation. There are quite a few log sources without a ready to use C API, but with a Python API. Using the Python source of syslog-ng you can leverage these.
https://www.syslog-ng.com/community/b/blog/posts/python-source-in-syslog-ng
Bulk mode message sending to Elasticsearch with syslog-ng http() destination ---------------------------------------------------------
Learn how to send log messages in bulk mode to your Elasticsearch server with syslog-ng. Bulk mode offers better performance, because it sends multiple log messages in a single POST request.
https://www.syslog-ng.com/community/b/blog/posts/bulk-mode-message-sending-t...
WEBINARS
You can watch our past webinars:
* Log ingestion to Splunk HEC: https://www.brighttalk.com/webcast/16207/338190
* High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943
Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Czanik, Péter
-
Pal, Laszlo
-
Scheidler, Balázs