- syslog-ng - lists.balabit.hu

[Bazsi's blog] LWN: syslog-ng rotten to the (Open) Core?
by Bazsi 08 Aug '10

08 Aug '10

This was first posted as a comment under an article on lwn.net, but I thought it was important enough to post it here for others not reading lwn. Please go ahead and read the original article which is about the "Open Core" business model and its problems from the Free Software community point of view. A commenter thought that syslog-ng was an example, which only exists as a marketing tool for the company's commercial offering. Anyway, here's my post: First of all, I want to make it clear that I'm biased on the syslog-ng case, but still wanted to express my opinion here. I'm biased as I'm the primary author of syslog-ng. I think syslog-ng is a completely different case from the one described by Neary. The GPL version is not crippleware, it was never published for marketing purposes only and for the majority of syslog-ng's existence only the Open Source stuff existed. The Premium Edition is only about 3 years old and syslog-ng started in 1998. We never removed features from the OSE version, the Premium Edition only included _additional_ features, and a lot of those are already available in the OSE. Some examples: * TLS support (became available in 3.0, almost 2 years ago) * SQL destination (became available in 2.1, 2.5 years ago) * performance improvements (3.0) * etc. In the other direction, we usually receive bugfixes and it is a pure technical reason that we used to require copyright assignment: I wanted to keep the two branches as close as possible (which if not done is the reason #1 why Open Core products become crippleware fast). _And_ since we heavily invested in automatic testing and our customers report bugs directly to us, we fix way more bugs in the OSE version than the community. But anyway, I didn't think that the dual license model was so problematic at the time we made this decision 3 years ago. Our efforts have never been "Rotten to the Open Core". If you don't believe that, check out the git repository or read the mailing list archive and see it yourself. And this whole mess is the past, OSE 3.2 has been relicensed, and it is true that we're going to publish non-free plugins, but anyone else is welcome to join and do the same. -- Posted By Bazsi to Bazsi's blog at 8/08/2010 01:23:00 PM

1 0

[Bazsi's blog] syslog-ng 3.2alpha2 released
by Bazsi 07 Aug '10

07 Aug '10

I've just uploaded syslog-ng 3.2alpha2 to the release directory. The last alpha release didn't compile on all supported platforms and the automatic test-suite was disabled, because it only worked if syslog-ng got installed first. These obstacles have been overcome and together with some fixes and a couple of new features, 3.2alpha2 is now available. I've also forward ported all bugfixes from syslog-ng 3.1.2. For those who are starting to experiment with the 3.2 branch, here's the list of new features compared to 3.1. Those who tried 3.2alpha1, the list of changes compared to 3.2alpha1 is at the end of this post. Since the documentation of syslog-ng is not yet up-to-date with the new features introduced, I've tried to also include URLs for the best known descriptions. The references may not be 100% accurate, but should give anyone interested an idea how to start experimenting. Also, please note that although this is an alpha release, the bulk of the changes are in the configuration parser, so once your configuration was parsed properly and syslog-ng starts up, an almost unchanged code is processing it. This means that this release should be good enough to start playing with. And feedback about what kind of syslog-ng.conf parsing errors you encounter on real-life configuration files is more than welcome. Code quality & functionality wise, this could be a beta release, I only expect "procedural" changes, like cleaning up the plugin names, which wouldn't be nice to do in a beta release (though not unheard of :) New features in 3.2: - Plugins: the new architecture replaces the old monolithic one, all syslog-ng functionality is loaded from external plugins when needed. It is possible to write plugins to extend syslog-ng functionality in the following areas: sources, destinations, filter expression, parsers, rewrite ops, message format. http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.ht… - The framework for a "syslog-ng configuration library" (aka SCL) a collection of configuration snippets installed along syslog-ng, simplifying the authoring of syslog-ng configuration files.http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=287993339599deac0442e26355c600b5aee63583 http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.ht… - pdbtool match is now able to read a file containing syslog messages and apply patterndb and a filter expression on the contents.http://bazsi.blogs.balabit.com/2010/07/patterndb-grep-on-steroids.… - pdbtool test is now able to perform pattern testing automatically based on the supplied example log message.http://marci.blogs.balabit.com/2010/07/pdbtool-test-and-pattern-dat… - Persistent state containing the current file position for file sources is now continously updated during runtime, instead of updating it only at exit, which makes it much more reliable in case syslog-ng doesn't terminate normally. - Better syntax error reporting in the configuration file.http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html - Support for reusable configuration snippets, similar to macros with parameters, named "blocks". http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-opened-experimental-blo… - Added a confgen plugin that includes the output of a program into the configuration file, making it possible to generate configuration file snippets dynamically. http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=5248ef6c49ff3af… - Support for BSD-style process accounting logs via the pacct() source driver defined in by SCL and the underlying pacctformat plugin.http://bazsi.blogs.balabit.com/2010/07/syslog-ng-and-process-account… - Support for explicit COMMITs in the SQL driver, this speeds up SQL INSERT rate significantly if flush_lines() is non-zero. http://bazsi.blogs.balabit.com/2010/04/explicit-transaction-support-in-sql.… - It is now possible to supply a filter to rewrite expressions and only apply the rewrite rule in case the filter matches.https://lists.balabit.hu/pipermail/syslog-ng/2010-July/014565.html - It is now possible to use multiple parser expressions in a single parser object, similar to rewrite rules. - Added support for using the include statement from anywhere in the configuration file, instead of only at top-level. Also introduced syslog-ng "global values" that can be defined and the substituted anywhere in the configuration file. http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=1203267c465256c… http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=52098762f27cde0… - Default configuration file supplied as part of SCL. Incompatible changes: - syslog-ng traditionally expected an optional hostname field even when a syslog message is received on a local transport (e.g. /dev/log). However no UNIX version is known to include this field. This caused problems when the application creating the log message has a space in its program name field. This behaviour has been changed for the unix-stream/unix-dgram/pipe drivers if the config version is 3.2 and can be restored by using an explicit 'expect-hostname' flag for the specific source. Changes since 3.2alpha1: - Now compiles on all platforms and the unit/functional tests also run. (tested: AIX, HP-UX, Solaris, FreeBSD, Linux, Tru64) - Fixed pdbtool match --debug-pattern output for ESTRING parsers. - Fixed a possible memory leak in the lexer, which would accumulate in case SIGHUPs. - Fixed Solaris STREAMS device support. - Forward ported all bugfixes from syslog-ng OSE 3.0 & 3.1 - Disable process accounting module by default as it doesn't compile on non-Linux platforms. - Added "pdbtool match --file" option to read and parse an existing logfile. - Added "pdbtool test" to check the log samples in the patterndb file. - Added "dont-create-tables" flag for the SQL destination to inhibit automatic table creation. - Added "condition()" support for rewrite expressions, which makes it possible to skip rewrite rules that do not match a filter expression. - Added "--module-path" command line option to control where modules are loaded from from the command line. Happy logging! -- Posted By Bazsi to Bazsi's blog at 8/07/2010 05:58:00 PM

1 0

[Bazsi's blog] syslog-ng name-value pair naming
by Bazsi 06 Aug '10

06 Aug '10

I was giving a lot of thought recently to the topic of naming name-value pairs in syslog-ng. Until now the only documented rule is stating somewhat vaguely that whenever you use a parser you should choose a name that has at least one dot in it, and this dot must not be the initial character. This means that names like MSG or .SDATA.meta.sequenceId are reserved for syslog-ng, and APACHE.CLIENT_IP is reserved for users. However things became more complex with syslog-ng OSE 3.2. Let's see what sources generate name-value pairs: - traditional macros (e.g. $DATE); these are not name-value pairs per-se, but behave much like them, except that they are read-only - syslog message fields (e.g. $MSG) if the message is coming from a syslog source - filters whenever the 'store-matches' flag is set and the regexp contains groups - rewrite rules, whenever the rewrite rule specifies a thus far unknown name-value pair, e.g. set("something" value("name-value.pair")); - and of course parsers when you tell syslog-ng to parse an input as a CSV, or use db-parser together with the patterns produced by the patterndb projectThe latest stuff generating name-value pairs is the support for process accounting logs, in this case even the syslog related fields are missing and only things like "pacct.ac_comm" (to contain the program name) are defined. So I was thinking whether it should be "pacct.ac_comm" or ".pacct.ac_comm". With the quoted rule it should be simple: it is generated by syslog-ng itself, thus it should be in the syslog-ng namespace and should start with a dot. However in the era of syslog-ng plugins, what consists of syslog-ng at all? First, I wanted to use "pacct.ac_comm" (e.g. without a dot), because I liked this name better. I was trying to explain myself why it would not violate the rule above. The explanation I had for myself was: I'm going to "register" names such as this in the patterndb SCHEMAS.txt file. With this - not yet published - explanation, I've committed a patch to convert the pacctformat plugin to use a dotless prefix. Next, I was figuring that it is true that process accounting creates name-value pairs without going through patternization, but I've felt, that nothing ensures that these name-value pairs would be directly usable, when trying to analyse the logs. The patterndb concept uses tags and schemas to convert the incoming unstructured data into a consistent structure. However, pacct may not completely match what the user needs. And, in the future, when SNMP traps or SQL table polling are going to be supported, it is going to be even more true: these name-value pairs may need a conversion: from the SNMP/pacct structure to the patterndb schema described structure in order to handle these message sources consistently with regular syslog (and to make it easy to correllate these). So at the end, I've committed another patch, this time going back to ".pacct" as a prefix and leaving the original naming rule intact. The "pacct" prefix is up to the users to use, they may want the same information in a "pacct" schema, but that may come from data not directly tied from process accounting (e.g. from syslog messages). So this post is about doing nothing with regards to the naming policy, but I thought it'd be important to shed a light behind the scenes. Giving such decisions enough thought and coming up a with a long-term plan makes our lives much easier in the future. This post may be a bit more involved than the others, but feel free to ask me to elaborate, if you are interested. -- Posted By Bazsi to Bazsi's blog at 8/06/2010 08:26:00 PM

1 0

greedy parser still broken?
by Jiann-Ming Su 06 Aug '10

06 Aug '10

I've reopened bug 87 (https://bugzilla.balabit.com/show_bug.cgi?id=87) because I think the greedy flag is still broken. I've updated the bug with my observations. Can someone else offer any more insight? Thanks.

2 4

Re: [syslog-ng] Strange behavior with the option "mark"
by Ilas, Yann 06 Aug '10

06 Aug '10

Thank you for your answer. When I start the syslog-ng with the option "mark" sets to 0, no MARKS messages are sent (as written in the OSE 3.1 Administrator Guide). This problem appears only when the option "mark" is set from a value <> 0 to a value = 0 and when I reload the configuration with a HUP signal. Regards, Yann I. On Tue, 2010-08-03 at 10:58 +0200, Ilas, Yann wrote: > Hello everybody, > > I'm currently using the application syslog-ng version 3.1.1 and there is > an odd behavior with the "mark" option. > > Here is the configuration file used for the test : > > @version: 3.0 > > options { > mark(7); > }; > > source s_local { internal(); unix-stream( "/dev/log" ); }; > > destination d_f_msg_unknown { > file ( /var/log/messages__unknown ); > }; > > log { > source (s_local); > destination (d_f_msg_unknown); > }; > > ...and I start the syslog-ng application like this : > # cd /opt/syslog-ng/sbin/ > # ./syslog-ng -Fevd --foreground --no-caps > --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK > --pidfile=/tmp/syslog-ng.conf.pid > > The output of the log file : > Aug 2 17:44:22 serveur01 -- MARK -- > Aug 2 17:44:29 serveur01 -- MARK -- > Aug 2 17:44:36 serveur01 -- MARK -- > (...) > > Then, I change the configuration file from "mark(7)" to "mark(0)" and > send a HUP signal between "17:44:36" and "17:44:43" > # kill -HUP $(cat /tmp/syslog-ng.conf.pid) > > Here is the output (tail -f /var/log/messages__unknown) : > Aug 2 17:44:36 serveur01 -- MARK -- > <HUP signal> > Aug 2 17:44:43 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > (...) > ...then the server syslog-ng sends a lot of "MARK" messages. I have to > stop the process with a "kill" or "ctrl+c". > > I have the same behaviour if I use "mark_freq" instead of "mark". > What's wrong with that option ? Did I miss something ? well, we should recognize mark(0) the same as mark(-1) effectively disabling the mark feature. right now, mark(0) means that there's zero time between two mark messages, effectively generating one mark message per poll iteration, this is what you see. Or, does this happen only if you do a SIGHUP? Or the same happens when you start syslog-ng? -- Bazsi

3 3

2 syslogng processes for the price of 1
by Manson, John 05 Aug '10

05 Aug '10

Good Day, I start syslog-ng, ver3.0.4, from /usr/local/sbin using the same script I run with older versions of ng on other servers but this is the first time I have seem 2 processes started. Ng is loading the correct conf file at /usr/local/etc/syslog-ng.conf and is writing log messages to the correct log files, no duplicate entries. Always 2 pids in sequence. If I kill the smaller pid, 3532 in this case, the ppid of 1 takes over for 3532 and ng, 3533, runs along just fine. Help? Thanks root 3532 1 0 15:09:54 pts/2 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid root 3533 3532 0 15:09:54 ? 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid John Manson CAO/HIR/NI/Data-Communications | U.S. House of Representatives | Washington, DC 20515 Desk: 202-226-4244 | Team: 202-225-5552 | john.manson(a)mail.house.gov

3 2

Debugging Pattern Match Failures
by Matthew Hall 04 Aug '10

04 Aug '10

Hello list, Recently I created a series of blasphemous scripts which convert some large collections of recorded log messages in my environment into pattern DB XML files. At first there were some syntax errors but I fixed all of these and the XML files are loading successfully. However I am running into some problems with the next step: getting the patterns to match against the incoming log messages. I suspect I am not properly stripping the headers off of the disk files of recorded messages I am using to generate the pattern DB XML files. I am wondering how I can enable the right debugging capabilities to get more detailed debug output from the pattern DB parser where I can see what strings are being processed so that I can fix this right instead of guessing repeatedly and incorrectly. Thanks, Matthew Hall.

3 12

syslog-ng 3.1.2 has been released
by devel＠balabit.hu 04 Aug '10

04 Aug '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.1.2 SUMMARY : new stable release DATE : Aug 4, 2010 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.1.2) has been released. For latest fixes in the 3.1.x feature branch you are recommended to upgrade to this version. CHANGES: 3.1.2 Mon, 02 Aug 2010 17:06:28 +0200 INCOMPATIBLE CHANGES: * This release changes the default for 'store-legacy-msghdr' flag for log sources. In previous releases this had to be enabled explicitly, because of the performance penalties, which do not apply to 3.1. The original behaviour can be restored by explicitly specifying 'dont-store-legacy-msghdr'. See this blog post for more details: http://bazsi.blogs.balabit.com/2010/05/small-incompatible-change-for-31.html Bugfixes: * When syslog-ng is reloaded, the local hostname value was not refreshed, causing syslog-ng to remember the hostname until the next restart. This may not play nice with DHCP configured hostnames, which may change dynamically. * When several SIGHUP signals are received in quick succession, the last one may have been dropped. This problem is fixed. * Make sure that numbers never get resolved using getpwnam/getgrnam (e.g. libnss functions), because this may cause deadlocks if the NSS provider is LDAP and the LDAP server is trying to log a message to syslog about invalid usernames. * Fixed flush_timeout() handling in for non-file destinations. Enabling flush_lines()/flush_timeout() for a non-file destination, either using a global option, or a per-destination option could cause excess CPU usage as long as the destination wasn't writable. * In case patterndb has a syntax error when reloading the file, syslog-ng automatically reverts to the old version instead of dropping pattern matching altogether. * Fixed pdbtool to properly handle accented characters in the command line. * Fixed the use of the greedy and drop-invalid flags for csv-parser. * Fixed a possible but rare memory leak in the handling of message suppression. Other changes: * Added "update-patterndb" command that merges patterndb files from ${sysconfdir}/patterndb.d and produces ${localstatedir}/patterndb.xml, the default file of the db-parser(). * Make it possible to specify the path to the syslog-ng control socket using a command line option in order to make it easier to launch multiple syslog-ng instances. * Introduced a new, cleaner syntax to specify that the permissions of a file are not to be changed. Use owner/group/perm options of a file without any parameters instead of "-1". The old mechanism had the problem that both per-file and global settings had to use "-1" to effectively stop syslog-ng from changing the uid/gid/perm values. The new syntax will make it possible to specify defaults in the global options section and customize those on a per-file basis. * syslog-ng will not go into the background if either the --syntax-only or --debug options are specified. Loggen: * Added SSL support. * Added the ability to read from a sample file Credits: * We would like to send our thank for all contributors sending us bugreports, documentation issues or patches. They are: * Marius Tomaschewski * Corey Hickey * Evan Rempel * Andy Kitchingman * andrewn at locus.net Hopefully noone was missed from this list. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Strange behavior with the option "mark"
by Ilas, Yann 03 Aug '10

03 Aug '10

Hello everybody, I'm currently using the application syslog-ng version 3.1.1 and there is an odd behavior with the "mark" option. Here is the configuration file used for the test : @version: 3.0 options { mark(7); }; source s_local { internal(); unix-stream( "/dev/log" ); }; destination d_f_msg_unknown { file ( /var/log/messages__unknown ); }; log { source (s_local); destination (d_f_msg_unknown); }; ...and I start the syslog-ng application like this : # cd /opt/syslog-ng/sbin/ # ./syslog-ng -Fevd --foreground --no-caps --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK --pidfile=/tmp/syslog-ng.conf.pid The output of the log file : Aug 2 17:44:22 serveur01 -- MARK -- Aug 2 17:44:29 serveur01 -- MARK -- Aug 2 17:44:36 serveur01 -- MARK -- (...) Then, I change the configuration file from "mark(7)" to "mark(0)" and send a HUP signal between "17:44:36" and "17:44:43" # kill -HUP $(cat /tmp/syslog-ng.conf.pid) Here is the output (tail -f /var/log/messages__unknown) : Aug 2 17:44:36 serveur01 -- MARK -- <HUP signal> Aug 2 17:44:43 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- (...) ...then the server syslog-ng sends a lot of "MARK" messages. I have to stop the process with a "kill" or "ctrl+c". I have the same behaviour if I use "mark_freq" instead of "mark". What's wrong with that option ? Did I miss something ? Regards, Yann I.

2 1

[Bug 84] New: Program destination blocks if program stops reading input
by bugzilla＠bugzilla.balabit.com 02 Aug '10

02 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=84 Summary: Program destination blocks if program stops reading input Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Syslog-ng OSE 3.0.5 and 3.0.6 - binary download from balabit and self compiled all behave the same. If the program in a destination stops reading its input, syslog-ng buffers a small number of messages and then stops reading the source (in my case a pipe). I would expect syslog-ng to read from its input until the log_fifo_size limit was reached and then drop messages. an strace is attached. The configuration file is -------------- options { log_fifo_size(1000000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; template t_application { template("$MESSAGE\n"); template_escape(no); }; template t_localsyslog { template("<$PRI>$S_ISODATE $HOST erempel: $MSGONLY\n"); template_escape(no); }; source local { pipe("/var/log/syslog.pipes/erempel" flags(no-parse) ); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) template(t_localsyslog) log_fifo_size(5000) ); }; destination erempel { program("/home1l/erempel/read-noread.pl" template(t_application) log_fifo_size(8000000) ); }; log { source(local); destination(erempel); }; log { source(int); destination(localsyslog); }; ---------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8