Dear syslog-ng users, I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways: 1) of course it has new features, see below for the most interesting bits 2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link: https://www.balabit.com/network-security/syslog-ng/opensource-logging-system... Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement. For those who hurry, here's a link for the source code: https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1bet... And here are the binaries for Linux/FreeBSD systems: https://www.balabit.com/network-security/syslog-ng/opensource-logging-system... Select the Downloads tab, and in the Version selector select 3.1beta1. What is new in syslog-ng OSE 3.1 -------------------------------- * Support for patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT. Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog. Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. Patterndb in general and the v1 format database is described in the syslog-ng manual at: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree: http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD The changes in the patterndb format as they evolved were described in Marton Illes's blog at http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html But see the other related posts as well. Old patterndb databases can be converted to the new format by putting them in a directory and using the pdbtool utility using the command: $ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml -D /opt/syslog-ng/etc/patterns.d Assuming the installation prefix of syslog-ng is /opt/syslog-ng Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location: https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/ You can convert these db files using pdbtool as described above. Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post. * Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message. See the manpage and Marci's post: http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html * Support for message tags: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering. http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html * Added support for rewriting structured data. Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message. Now all these became possible by using the same syntax that didn't work earlier, e.g. rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); }; * Macros and name-value pairs got a little tighter integration, in filters where syslog-ng 3.0 was limited to only use name-value pairs, with 3.1 you can also use macros. The following now works: match("<regexp>" value("R_DATE")); syslog-ng is now warning you in case you are using '$' prefix in the value syntax. * Enhanced dynamic name-value performance by a factor of three. The summary says it all, the performance dynamic name-value pairs that the various parsers produce got faster, thus the performance penalty of structuring the incoming messages got smaller. * Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These are patterndb parsers to make it easier to describe log messages. * Added non-standard and non-portable facility codes (range 10-15), decouple syslog-ng facility name information from the system used to compile syslog-ng on. Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. Any feedback, success/failure reports is more than appreciated. -- Bazsi