[syslog-ng] poll: web interface for syslog-ng

[John Kristoff]

On Wed, 25 Nov 2020 14:46:44 +0000
"Peter Czanik (pczanik)" <Peter.Czanik at oneidentity.com> wrote:

> Which web interface do you use to search / analyze / alert on your
> logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still
> grep/awk?

Hi Peter,

I'm been using syslog-ng since early to mid 2000's, mainly as a
replacement for stock syslog and more recently rsyslog.  A few years
back I started down the path of trying Elasticsearch, but never
finished getting it up and running.  At the time it was too much
trouble for what was just tinkering at the time.  Maybe one day I'll
give it another try, but I don't have a large need right now.

It isn't all just grep/awk and for me however.  Originally, one of my
first uses was on Solaris where I found syslog-ng to be vastly more
efficient (less CPU for the workload) than the stock daemon.  I grew to
appreciate what now seem like simple things, the macros to put logs in
paths and file names expanded by date for instance.  Later I used this
feature to create circular logs when storage space was important and
history wasn't (just use the day of the week, overwriting each day's
file as you go).

My appreciation and usage grew further when I used it as the key
component of a security sensor network.  Comparing it to rsyslog at the
time, when it was replacing syslog-ng as the default daemon in distros,
I found rsyslog to buggy and the documentation very poor in comparison.
Things have probably changed for rsyslog for the better now, but I've
never looked back.  Back then I started making heavy use of the the
pattern matching database feature, which was quite novel then, and
database integration.

Those two features, coupled with TLS support I was able to remotely log
from dozens of systems all over the world to a central collector and
build up my customer tables. Data was logged in files for archival, but
the things I cared about I parsed and stuck into the database.  This
project has continued today and I'm still running over 100 nodes this
way providing various kinds of feeds to the security community. You can
read more about this project in my article from USENIX's ;login: Fall
2018 issue or visit the homepage of the domain from which I'm sending
this email.

In my network role at a university we have Splunk where some logs go,
but I personally still to work with raw logs since I rarely need to
look at them, and when I do it is often to conduct some very
specific aggregate analysis or exploration that Splunk won't do with
more work.  I'd just prefer to use unix tools and raw data for those
rare occasions.

So yes grep/awk (and perl, python, sort, cut and so on), but for my
DataPlane project it is often SQL queries or custom code to create
reports from the database that received data directly from syslog-ng.

My standard system build is to always replace whatever syslog daemon
might be installed with syslog-ng, change the default timestamp settings
to ts_format(iso) then add other customizations into conf.d/ as I go.

Hope that helps, sorry not much for a web interface user here.

John