poll: web interface for syslog-ng
Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sys...) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513 Thanks, Peter Peter Czanik (CzP) <peter.czanik@oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik
On Wed, 25 Nov 2020 14:46:44 +0000 "Peter Czanik (pczanik)" <Peter.Czanik@oneidentity.com> wrote:
Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk?
Hi Peter, I'm been using syslog-ng since early to mid 2000's, mainly as a replacement for stock syslog and more recently rsyslog. A few years back I started down the path of trying Elasticsearch, but never finished getting it up and running. At the time it was too much trouble for what was just tinkering at the time. Maybe one day I'll give it another try, but I don't have a large need right now. It isn't all just grep/awk and for me however. Originally, one of my first uses was on Solaris where I found syslog-ng to be vastly more efficient (less CPU for the workload) than the stock daemon. I grew to appreciate what now seem like simple things, the macros to put logs in paths and file names expanded by date for instance. Later I used this feature to create circular logs when storage space was important and history wasn't (just use the day of the week, overwriting each day's file as you go). My appreciation and usage grew further when I used it as the key component of a security sensor network. Comparing it to rsyslog at the time, when it was replacing syslog-ng as the default daemon in distros, I found rsyslog to buggy and the documentation very poor in comparison. Things have probably changed for rsyslog for the better now, but I've never looked back. Back then I started making heavy use of the the pattern matching database feature, which was quite novel then, and database integration. Those two features, coupled with TLS support I was able to remotely log from dozens of systems all over the world to a central collector and build up my customer tables. Data was logged in files for archival, but the things I cared about I parsed and stuck into the database. This project has continued today and I'm still running over 100 nodes this way providing various kinds of feeds to the security community. You can read more about this project in my article from USENIX's ;login: Fall 2018 issue or visit the homepage of the domain from which I'm sending this email. In my network role at a university we have Splunk where some logs go, but I personally still to work with raw logs since I rarely need to look at them, and when I do it is often to conduct some very specific aggregate analysis or exploration that Splunk won't do with more work. I'd just prefer to use unix tools and raw data for those rare occasions. So yes grep/awk (and perl, python, sort, cut and so on), but for my DataPlane project it is often SQL queries or custom code to create reports from the database that received data directly from syslog-ng. My standard system build is to always replace whatever syslog daemon might be installed with syslog-ng, change the default timestamp settings to ts_format(iso) then add other customizations into conf.d/ as I go. Hope that helps, sorry not much for a web interface user here. John
Thanks for the detailed response. I was thrilled while reading your email and this confirms that my ongoing efforts were worth it in the last 22 years :) Thanks Bazsi On Wed, Nov 25, 2020 at 5:03 PM John Kristoff <jtk@dataplane.org> wrote:
On Wed, 25 Nov 2020 14:46:44 +0000 "Peter Czanik (pczanik)" <Peter.Czanik@oneidentity.com> wrote:
Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk?
Hi Peter,
I'm been using syslog-ng since early to mid 2000's, mainly as a replacement for stock syslog and more recently rsyslog. A few years back I started down the path of trying Elasticsearch, but never finished getting it up and running. At the time it was too much trouble for what was just tinkering at the time. Maybe one day I'll give it another try, but I don't have a large need right now.
It isn't all just grep/awk and for me however. Originally, one of my first uses was on Solaris where I found syslog-ng to be vastly more efficient (less CPU for the workload) than the stock daemon. I grew to appreciate what now seem like simple things, the macros to put logs in paths and file names expanded by date for instance. Later I used this feature to create circular logs when storage space was important and history wasn't (just use the day of the week, overwriting each day's file as you go).
My appreciation and usage grew further when I used it as the key component of a security sensor network. Comparing it to rsyslog at the time, when it was replacing syslog-ng as the default daemon in distros, I found rsyslog to buggy and the documentation very poor in comparison. Things have probably changed for rsyslog for the better now, but I've never looked back. Back then I started making heavy use of the the pattern matching database feature, which was quite novel then, and database integration.
Those two features, coupled with TLS support I was able to remotely log from dozens of systems all over the world to a central collector and build up my customer tables. Data was logged in files for archival, but the things I cared about I parsed and stuck into the database. This project has continued today and I'm still running over 100 nodes this way providing various kinds of feeds to the security community. You can read more about this project in my article from USENIX's ;login: Fall 2018 issue or visit the homepage of the domain from which I'm sending this email.
In my network role at a university we have Splunk where some logs go, but I personally still to work with raw logs since I rarely need to look at them, and when I do it is often to conduct some very specific aggregate analysis or exploration that Splunk won't do with more work. I'd just prefer to use unix tools and raw data for those rare occasions.
So yes grep/awk (and perl, python, sort, cut and so on), but for my DataPlane project it is often SQL queries or custom code to create reports from the database that received data directly from syslog-ng.
My standard system build is to always replace whatever syslog daemon might be installed with syslog-ng, change the default timestamp settings to ts_format(iso) then add other customizations into conf.d/ as I go.
Hope that helps, sorry not much for a web interface user here.
John
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
grep awk cheers Matt Von: "Peter Czanik (pczanik)" <Peter.Czanik@oneidentity.com> An: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu> Datum: 25.11.2020 15:46 Betreff: [syslog-ng] poll: web interface for syslog-ng Gesendet von: "syslog-ng" <syslog-ng-bounces@lists.balabit.hu> Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out ( https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sys... ) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513 Thanks, Peter Peter Czanik (CzP) <peter.czanik@oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq Persönlich haftende Gesellschafter: Harald Illy, Friedrich von Metzler, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 27 515 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.
participants (4)
-
Balazs Scheidler
-
John Kristoff
-
Matthias Gruber
-
Peter Czanik (pczanik)