[syslog-ng] syslog-ng OSE git tree opened

[Balazs Scheidler]

Dear syslog-ng users,

I'm proud to announce that I've opened the syslog-ng OSE 3.0 branch and
pushed a backport of the result of the latest Premium Edition
development cycle.

The size of the patch is enourmous, it took me a while to integrate it
back to OSE 3.0, that's why this announcement is coming so late. The
diffstat is as follows:

 175 files changed, 21537 insertions(+), 7377 deletions(-)

The list of changes of interest are:
  * various performance improvements, syslog-ng can do about 120-130k 
    messages/sec in case no destination is written to (not very useful though), 
    and about 75k msg/sec when writing to a plain text file

  * TLS encrypted channels, works the same as the PE version, documentation is:
    http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch03s09.html

  * support for the new IETF syslog protocols, in order to use this, you need to
    use the syslog() driver either as a source or a destination. It behaves quiet 
    closely to tcp/udp, except it uses the new protocol
    
  * log message classification using a database, scales up to tens of thousands 
    of messages per second, the pattern database also serves as a means to 
    parse variable parts of messages. No real documentation so far, there's an 
    XML Schema file in the tarball, it can be used using the db_parser() parser.

  * rewrite parts of the message using regexps, and simple string search and replace. Here's
    a post on the syslog-ng mailing list that has some info on usage:
    http://marc.info/?l=syslog-ng&m=122051372205555&w=2

  * parse parts of the syslog message, makes 
    it possible to parse fields in an Apache log for instance. Also the parsed values can
    be used as macros. Again, the post above has some info:
    http://marc.info/?l=syslog-ng&m=122051372205555&w=2
    
  * generalize the log message paths, instead of a flat list of source->destination 
    connections, log statements can be organized into a tree, making the 
    configuration simpler and the performance better, here is a description:
    http://bazsi.blogs.balabit.com/2008/02/redesigning-syslog-ng-internals.html

  * PCRE and glob filters (in addition to POSIX regexps), use the type("pcre") 
    option in the filter, like this: host("something" type("pcre"))

  * support for capabilities under Linux, this running syslog-ng as non-root is possible,
    also with reload support, see the documentation of Linux capabilities in capabilities(7),
    for the syntax of the --caps option, see cap_from_text(3)

  * program sources, simply use program("cat /dev/xconsole") in your configuration, the program
    will be restarted automatically if that exits.

  * syslog-ng got the ability to restart itself if it crashes, see the --process-mode 
    command line option

  * new statistics framework that can be queried using UNIX domain sockets, provides 
    per-source-host and per-destination-host statistics. In order to use it, possibly set 
    the stats_level() option and write "STATS\n" to /var/run/syslog-ng.ctl which is a UNIX domain socket.

This mass amount of changes also mean some incompatibilities between 2.1 and 3.0, however 
3.0 will work in compatibility mode unless it is told to use the 3.0 configuration 
file format. It'll also report warnings against your configuration file you should change
in order to use all the features and best performance. 

The most important incompatibilities include:
  * the $MSG macro will not include the "program[pid]:" string anymore, you should prepent
    an $MSGHDR macro in order to have that, in 2.1 mode the MSG will expand as before, but 
    please note that this value is calculated everytime you reference the macro, thus it
    affects performance.
  * some defaults were changed (chain_hostname, template-escape, follow-freq for file sources),
    in 2.1 mode the old defaults are used but a warning is emitted
  * the input path does not remove NL characters anymore by default, but you can ask for this 
    behaviour by specifying the "no-multi-line" flag.

The documentation for the new features is not yet completely finished, I
hope to get a snapshot of the current state to the website soon. In the
meanwhile either use the source :) or ask direct questions on the
mailing list. I'll also try to make some posts on my blog about various
features in more detail and configuration examples.

-- 
Bazsi