[syslog-ng] syslog-ng OSE 3.1beta1 released

[Balazs Scheidler]

Dear syslog-ng users,

I'm proud to announce that syslog-ng OSE 3.1 has been released and
uploaded to our webserver. This version is new in two ways:

1) of course it has new features, see below for the most interesting
bits

2) it is a "feature release", which means that once syslog-ng 3.2 or
syslog-ng 4.0 is released, the support for this release will be ceased.
See our new version policy at this link:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap.bbx

Since the documentation is not yet up to date with this beta release,
I'll try to include the most crucial information about the new features
right here in this announcement.

For those who hurry, here's a link for the source code:

https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1beta1/source/syslog-ng_3.1beta1.tar.gz

And here are the binaries for Linux/FreeBSD systems:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/

Select the Downloads tab, and in the Version selector select 3.1beta1.

What is new in syslog-ng OSE 3.1
--------------------------------

* Support for patterndb v3 format, along with a bunch of new
  parsers: ANYSTRING, IPv6, IPvANY and FLOAT.

  Patterndb (more exactly the db-parser()) is a high performance message 
  classifier and information extraction tool, that makes it easy to get away
  from the unstructured nature of syslog.

  Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at 
  the 3rd iteration, hopefully slowly reaching its final form.

  Patterndb in general and the v1 format database is described in the syslog-ng 
  manual at:
  
  http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html

  The XML schemas that describe the different patterndb versions are available in
  the syslog-ng source tree:

  http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD

  The changes in the patterndb format as they evolved were described in Marton Illes's blog at
  
  http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html

  But see the other related posts as well.
 
  Old patterndb databases can be converted to the new format by putting them
  in a directory and using the pdbtool utility using the command:

  $ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml -D /opt/syslog-ng/etc/patterns.d

  Assuming the installation prefix of syslog-ng is /opt/syslog-ng

  Some v2 format patterns are distributed by BalaBit itself for its SSB product, 
  download location:

  https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/

  You can convert these db files using pdbtool as described above.

  Work is ongoing to publish a more comprehensive patterndb, but more on that
  in a separate post.

* Added a new "pdbtool" utility to manage patterndb files: convert
  them from v1 or v2 format, merge mulitple patterndb files into one
  and look up matching patterns given a specific message.

  See the manpage and Marci's post:

  http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html

* Support for message tags: tags can be assigned to log messages as
  they enter syslog-ng: either by the source driver or via patterndb.
  Later it these tags can be used for efficient filtering.

  http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html

* Added support for rewriting structured data.

  Earlier structured data fields in the new RFC5424 style syslog protocol
  were only read-only values that could be referenced in a template, but they 
  couldn't be changed, and neither was it possible to add new fields in an 
  already existing syslog message.

  Now all these became possible by using the same syntax that didn't work 
  earlier, e.g.

  rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); };

* Macros and name-value pairs got a little tighter integration,
  in filters where syslog-ng 3.0 was limited to only use name-value
  pairs, with 3.1 you can also use macros.

  The following now works:

  match("<regexp>" value("R_DATE"));

  syslog-ng is now warning you in case you are using '$' prefix in 
  the value syntax.

* Enhanced dynamic name-value performance by a factor of three.

  The summary says it all, the performance dynamic name-value pairs that
  the various parsers produce got faster, thus the performance penalty of
  structuring the incoming messages got smaller.

* Some parsers got additional features: NUMBER is now able to parse
  hexadecimal numbers, ESTRING is now able to search for a sequence
  of characters as the end of the string.

  These are patterndb parsers to make it easier to describe log messages.

* Added non-standard and non-portable facility codes (range 10-15),
  decouple syslog-ng facility name information from the system used
  to compile syslog-ng on.

  Until this time the facility codes as understood by syslog-ng were
  dependant on the platform syslog-ng was compiled on. This is not true
  anymore, syslog-ng comes with its own "facility" code assignments, based
  on the RFC, and adding some non-standard values found on various 
  UNIX systems.

Any feedback, success/failure reports is more than appreciated.

-- 
Bazsi