Dear syslog-ng users, I'm proud to announce that I've opened the syslog-ng OSE 3.0 branch and pushed a backport of the result of the latest Premium Edition development cycle. The size of the patch is enourmous, it took me a while to integrate it back to OSE 3.0, that's why this announcement is coming so late. The diffstat is as follows: 175 files changed, 21537 insertions(+), 7377 deletions(-) The list of changes of interest are: * various performance improvements, syslog-ng can do about 120-130k messages/sec in case no destination is written to (not very useful though), and about 75k msg/sec when writing to a plain text file * TLS encrypted channels, works the same as the PE version, documentation is: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch03s09.html * support for the new IETF syslog protocols, in order to use this, you need to use the syslog() driver either as a source or a destination. It behaves quiet closely to tcp/udp, except it uses the new protocol * log message classification using a database, scales up to tens of thousands of messages per second, the pattern database also serves as a means to parse variable parts of messages. No real documentation so far, there's an XML Schema file in the tarball, it can be used using the db_parser() parser. * rewrite parts of the message using regexps, and simple string search and replace. Here's a post on the syslog-ng mailing list that has some info on usage: http://marc.info/?l=syslog-ng&m=122051372205555&w=2 * parse parts of the syslog message, makes it possible to parse fields in an Apache log for instance. Also the parsed values can be used as macros. Again, the post above has some info: http://marc.info/?l=syslog-ng&m=122051372205555&w=2 * generalize the log message paths, instead of a flat list of source->destination connections, log statements can be organized into a tree, making the configuration simpler and the performance better, here is a description: http://bazsi.blogs.balabit.com/2008/02/redesigning-syslog-ng-internals.html * PCRE and glob filters (in addition to POSIX regexps), use the type("pcre") option in the filter, like this: host("something" type("pcre")) * support for capabilities under Linux, this running syslog-ng as non-root is possible, also with reload support, see the documentation of Linux capabilities in capabilities(7), for the syntax of the --caps option, see cap_from_text(3) * program sources, simply use program("cat /dev/xconsole") in your configuration, the program will be restarted automatically if that exits. * syslog-ng got the ability to restart itself if it crashes, see the --process-mode command line option * new statistics framework that can be queried using UNIX domain sockets, provides per-source-host and per-destination-host statistics. In order to use it, possibly set the stats_level() option and write "STATS\n" to /var/run/syslog-ng.ctl which is a UNIX domain socket. This mass amount of changes also mean some incompatibilities between 2.1 and 3.0, however 3.0 will work in compatibility mode unless it is told to use the 3.0 configuration file format. It'll also report warnings against your configuration file you should change in order to use all the features and best performance. The most important incompatibilities include: * the $MSG macro will not include the "program[pid]:" string anymore, you should prepent an $MSGHDR macro in order to have that, in 2.1 mode the MSG will expand as before, but please note that this value is calculated everytime you reference the macro, thus it affects performance. * some defaults were changed (chain_hostname, template-escape, follow-freq for file sources), in 2.1 mode the old defaults are used but a warning is emitted * the input path does not remove NL characters anymore by default, but you can ask for this behaviour by specifying the "no-multi-line" flag. The documentation for the new features is not yet completely finished, I hope to get a snapshot of the current state to the website soon. In the meanwhile either use the source :) or ask direct questions on the mailing list. I'll also try to make some posts on my blog about various features in more detail and configuration examples. -- Bazsi