[syslog-ng] [announcement] syslog-ng contributions redefined

[Balazs Scheidler]

Dear syslog-ng users,

This is a repost of my blog post at http://bazsi.blogs.balabit.com/
which I felt was important enough to repeat here. Links may not be
properly reproduced, so please follow the link above if you'd like to
browse the references as well.


syslog-ng has been around for about 12 years now, but I think the
biggest change in the project's life is imminent: with the upcoming
release of syslog-ng OSE 3.2, syslog-ng will become an independent
entity.

Until now, syslog-ng was primarily maintained & developed by BalaBit,
copyrights needed to be reassigned in order to grant BalaBit special
privileges. BalaBit used her privileges to create a dual-licensed fork
of syslog-ng, named "syslog-ng Premium Edition". The value we offer over
the Open Source Edition of syslog-ng are things that larger enterprises
require:

      * support on a large number of UNIX platforms (27 as of 3.1), 
      * smaller and larger feature differences (like the
        encrypted/digitally signed logfile feature)
      * better test coverage and release management
      * longer term support

Although perfectly legal, this business model was not welcome in various
Free Software communities, and has caused friction and harm, because
BalaBit has enjoyed a privilege that no others could get. We plan to fix
this situation and we've worked hard in the past months to make this
possible.

We're letting syslog-ng go: no signed Contributory License Agreements
will be required in order to contribute to syslog-ng in the future.

This is not just a matter of policy: while BalaBit wants to be a true
citizen of the Free Software world, we also need to ensure the continued
revenue stream that the Premium Edition provides. The adjusted business
model allows us to deliver what our customers need, and at the same time
make it possible for anyone else in the community to have the same
privileges as BalaBit has.

The syslog-ng changes in 3.2 that I feel are most important are detailed
below.

Plugins

syslog-ng was transformed from being a monolithic executable, to a core
and a set of plugins.

Plugins range from source and destination drivers, filters, parsers,
rewrite objects, anything can be extended with the use of a relatively
simple plugin.

And this change didn't transform the configuration file format, it
remained the same as before: readable and flexible. If you don't want to
look under the hood, no functionality has changed, except plugins are
loaded at runtime, and as a side effect, the syntax error reports are
way better.

The core is licensed under the LGPL and plugins are licensed under the
GPL. This legal framework allows BalaBit to deliver value to its
customers in non-free plugins and syslog-ng related services.

syslog-ng Configuration Library

syslog-ng is an infrastructure element: very flexible, but sometimes
intimidating. For example, having to specify the pad_size() option for
an HP-UX /dev/log device explicitly makes syslog-ng flexible, but not
very user-friendly.

It is common to share working configuration snippets on the mailing list
or elsewhere in the syslog-ng community. We try to concentrate this
effort with the creation of the syslog-ng Configuration Library (aka
SCL). This library is a set of config snippets that can be included into
a syslog-ng configuration file, using proper defaults but still allowing
customization.

For example: we've created a source driver named "system()" which
automatically expands to the local log devices as needed by the current
Operating System syslog-ng runs on, but it is also possible to create
application specific configuration snippets (for example apache source?)
and ship it as a ready to use config block. See this post for more
information.

This makes it possible to create a configuration file that will run on
each of your platforms. In fact, we did that too: syslog-ng now comes
with a default configuration file.

Support for non-syslog messages

Since syslog-ng is now plugin based and even the "syslog" message format
as such is a plugin, it is now quite easy to add support for non-syslog
message sources. As an example, we've added a plugin to parse Linux
process accounting records, which makes it trivial to collect this data
as well and possibly use it as a source of information when correllating
data.

Future plugins like creating a MIB aware SNMP listener, or possibly
processing netflow data, but I'd like to create a generic SQL source as
well.

Support for these formats doesn't mean that syslog-ng would transform
them into textual messages: proper name-value structure is kept and it
is possible to put them into nicely structured SQL tables. Of course if
all you want to transform them to text for readability, that's possible
too with the use of a template().

patterndb project

Like I announced before, we're starting a parallel project to create a
set of message patterns directly usable with syslog-ng's db-parser().

Current state

The changes are happening in the syslog-ng 3.2 repository at:

http://git.balabit.hu/

We have created a source-only 3.2alpha1 release of the current state. It
runs our automated test harness on Linux, but of course it is not yet
recommended in production environments. The release can be downloaded
from www.balabit.com.

There were significant changes in how syslog-ng is compiled, thus it is
expected that we have build issues on non-Linux systems. We expect to
address these issues as they are found and create a 3.2beta1 release,
for a longer beta testing.

Future

Parallel to fixing up the remaining issues on syslog-ng 3.2, we're going
to open new ways to improve syslog-ng in the 3.3 branch. The most
important item on our product backlog is to address the scalability
problems and improve performance of syslog-ng.

Summary

With the upcoming changes in syslog-ng OSE 3.2 possible contributions
will be greatly expanded:
      * write a pattern for your favourite application and process log
        data faster + easier
      * write an SCL configuration snippet, make your application easier
        to integrate with syslog-ng
      * write a plugin for your favourite NoSQL database,
      * write a plugin for a transformation that syslog-ng is not
        capable of doing right now (what about facility / severity
        rewrite rules? easy as a piece of cake)
      * write a plugin for things that you do with an external script
      * or contribute to the core of syslog-ng.

And all this without having to sign paperwork...

-- 
Bazsi