- syslog-ng - lists.balabit.hu

Using source tags in rewrites?
by Steve Barnes 14 Aug '10

14 Aug '10

Hi Is it possible to use source tags in rewrite rules? The documentation indicates only macros are acceptable? Using Syslog 3.1.2. Cheers Steve

4 7

[Bug 87] New: csv-parser: specifying drop-invalid and greedy flags always fails.
by bugzilla＠bugzilla.balabit.com 14 Aug '10

14 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=87 Summary: csv-parser: specifying drop-invalid and greedy flags always fails. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: alex-pub.balabit(a)reflexion.net Type of the Report: bug Estimated Hours: 0.0 In this example, I am trying to parse the postfix log statement. Postfix specifies $PROGRAM as a pair separated by slash, e.g "postfix/master" or "postfix/postfix-script". The "postfix" part is configurable, and I would like to parse it out from the $PROGRAM macro. I tried it first with this parser parser p_rfx_postfix_program_parser { csv-parser( columns( "POSTFIX.SERVICE_NAME", "POSTFIX.PROGRAM" ) delimiters( "/" ) flags( drop-invalid, escape-none, greedy ) template( "${PROGRAM}" ) ); }; The intention here is that ${PROGRAM} should always contain at least 1 forward slash, and I only want to parse the part before the first slash. However this parser always fails. Removing greedy flag makes this parser succeed, however it does change the meaning of the action that I am trying to perform. Looking at the logparser.c: log_csv_parser_process, I've noticed that when LOG_CSV_PARSER_GREEDY flag is on, then `cur_column` variable is reset to NULL. Then right before the function ends it checks if `cur_column` is null and LOG_CSV_PARSER_DROP_INVALID flag is on, then fails the function. I think the check at the end of the function should take into account whether GREEDY operation has been performed or not. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bazsi's blog] blog moved
by Bazsi 13 Aug '10

13 Aug '10

I'm moving this blog to a wordpress instance deployed on our company webserver. The URL should be unchanged but the RSS/Atom feeds have changed, so please update your bookmarks. Thanks and sorry for the confusion. -- Posted By Bazsi to Bazsi's blog at 8/12/2010 03:23:00 PM

1 0

Using source tags in rewrites?
by Steve Barnes 10 Aug '10

10 Aug '10

Hi Is it possible to use source tags in rewrite rules? The documentation indicates only macros are acceptable? Cheers Steve

1 0

[Bazsi's blog] LWN: syslog-ng rotten to the (Open) Core?
by Bazsi 08 Aug '10

08 Aug '10

This was first posted as a comment under an article on lwn.net, but I thought it was important enough to post it here for others not reading lwn. Please go ahead and read the original article which is about the "Open Core" business model and its problems from the Free Software community point of view. A commenter thought that syslog-ng was an example, which only exists as a marketing tool for the company's commercial offering. Anyway, here's my post: First of all, I want to make it clear that I'm biased on the syslog-ng case, but still wanted to express my opinion here. I'm biased as I'm the primary author of syslog-ng. I think syslog-ng is a completely different case from the one described by Neary. The GPL version is not crippleware, it was never published for marketing purposes only and for the majority of syslog-ng's existence only the Open Source stuff existed. The Premium Edition is only about 3 years old and syslog-ng started in 1998. We never removed features from the OSE version, the Premium Edition only included _additional_ features, and a lot of those are already available in the OSE. Some examples: * TLS support (became available in 3.0, almost 2 years ago) * SQL destination (became available in 2.1, 2.5 years ago) * performance improvements (3.0) * etc. In the other direction, we usually receive bugfixes and it is a pure technical reason that we used to require copyright assignment: I wanted to keep the two branches as close as possible (which if not done is the reason #1 why Open Core products become crippleware fast). _And_ since we heavily invested in automatic testing and our customers report bugs directly to us, we fix way more bugs in the OSE version than the community. But anyway, I didn't think that the dual license model was so problematic at the time we made this decision 3 years ago. Our efforts have never been "Rotten to the Open Core". If you don't believe that, check out the git repository or read the mailing list archive and see it yourself. And this whole mess is the past, OSE 3.2 has been relicensed, and it is true that we're going to publish non-free plugins, but anyone else is welcome to join and do the same. -- Posted By Bazsi to Bazsi's blog at 8/08/2010 01:23:00 PM

1 0

[Bazsi's blog] syslog-ng 3.2alpha2 released
by Bazsi 07 Aug '10

07 Aug '10

I've just uploaded syslog-ng 3.2alpha2 to the release directory. The last alpha release didn't compile on all supported platforms and the automatic test-suite was disabled, because it only worked if syslog-ng got installed first. These obstacles have been overcome and together with some fixes and a couple of new features, 3.2alpha2 is now available. I've also forward ported all bugfixes from syslog-ng 3.1.2. For those who are starting to experiment with the 3.2 branch, here's the list of new features compared to 3.1. Those who tried 3.2alpha1, the list of changes compared to 3.2alpha1 is at the end of this post. Since the documentation of syslog-ng is not yet up-to-date with the new features introduced, I've tried to also include URLs for the best known descriptions. The references may not be 100% accurate, but should give anyone interested an idea how to start experimenting. Also, please note that although this is an alpha release, the bulk of the changes are in the configuration parser, so once your configuration was parsed properly and syslog-ng starts up, an almost unchanged code is processing it. This means that this release should be good enough to start playing with. And feedback about what kind of syslog-ng.conf parsing errors you encounter on real-life configuration files is more than welcome. Code quality & functionality wise, this could be a beta release, I only expect "procedural" changes, like cleaning up the plugin names, which wouldn't be nice to do in a beta release (though not unheard of :) New features in 3.2: - Plugins: the new architecture replaces the old monolithic one, all syslog-ng functionality is loaded from external plugins when needed. It is possible to write plugins to extend syslog-ng functionality in the following areas: sources, destinations, filter expression, parsers, rewrite ops, message format. http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.ht… - The framework for a "syslog-ng configuration library" (aka SCL) a collection of configuration snippets installed along syslog-ng, simplifying the authoring of syslog-ng configuration files.http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=287993339599deac0442e26355c600b5aee63583 http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.ht… - pdbtool match is now able to read a file containing syslog messages and apply patterndb and a filter expression on the contents.http://bazsi.blogs.balabit.com/2010/07/patterndb-grep-on-steroids.… - pdbtool test is now able to perform pattern testing automatically based on the supplied example log message.http://marci.blogs.balabit.com/2010/07/pdbtool-test-and-pattern-dat… - Persistent state containing the current file position for file sources is now continously updated during runtime, instead of updating it only at exit, which makes it much more reliable in case syslog-ng doesn't terminate normally. - Better syntax error reporting in the configuration file.http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html - Support for reusable configuration snippets, similar to macros with parameters, named "blocks". http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-opened-experimental-blo… - Added a confgen plugin that includes the output of a program into the configuration file, making it possible to generate configuration file snippets dynamically. http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=5248ef6c49ff3af… - Support for BSD-style process accounting logs via the pacct() source driver defined in by SCL and the underlying pacctformat plugin.http://bazsi.blogs.balabit.com/2010/07/syslog-ng-and-process-account… - Support for explicit COMMITs in the SQL driver, this speeds up SQL INSERT rate significantly if flush_lines() is non-zero. http://bazsi.blogs.balabit.com/2010/04/explicit-transaction-support-in-sql.… - It is now possible to supply a filter to rewrite expressions and only apply the rewrite rule in case the filter matches.https://lists.balabit.hu/pipermail/syslog-ng/2010-July/014565.html - It is now possible to use multiple parser expressions in a single parser object, similar to rewrite rules. - Added support for using the include statement from anywhere in the configuration file, instead of only at top-level. Also introduced syslog-ng "global values" that can be defined and the substituted anywhere in the configuration file. http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=1203267c465256c… http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=52098762f27cde0… - Default configuration file supplied as part of SCL. Incompatible changes: - syslog-ng traditionally expected an optional hostname field even when a syslog message is received on a local transport (e.g. /dev/log). However no UNIX version is known to include this field. This caused problems when the application creating the log message has a space in its program name field. This behaviour has been changed for the unix-stream/unix-dgram/pipe drivers if the config version is 3.2 and can be restored by using an explicit 'expect-hostname' flag for the specific source. Changes since 3.2alpha1: - Now compiles on all platforms and the unit/functional tests also run. (tested: AIX, HP-UX, Solaris, FreeBSD, Linux, Tru64) - Fixed pdbtool match --debug-pattern output for ESTRING parsers. - Fixed a possible memory leak in the lexer, which would accumulate in case SIGHUPs. - Fixed Solaris STREAMS device support. - Forward ported all bugfixes from syslog-ng OSE 3.0 & 3.1 - Disable process accounting module by default as it doesn't compile on non-Linux platforms. - Added "pdbtool match --file" option to read and parse an existing logfile. - Added "pdbtool test" to check the log samples in the patterndb file. - Added "dont-create-tables" flag for the SQL destination to inhibit automatic table creation. - Added "condition()" support for rewrite expressions, which makes it possible to skip rewrite rules that do not match a filter expression. - Added "--module-path" command line option to control where modules are loaded from from the command line. Happy logging! -- Posted By Bazsi to Bazsi's blog at 8/07/2010 05:58:00 PM

1 0

[Bazsi's blog] syslog-ng name-value pair naming
by Bazsi 06 Aug '10

06 Aug '10

I was giving a lot of thought recently to the topic of naming name-value pairs in syslog-ng. Until now the only documented rule is stating somewhat vaguely that whenever you use a parser you should choose a name that has at least one dot in it, and this dot must not be the initial character. This means that names like MSG or .SDATA.meta.sequenceId are reserved for syslog-ng, and APACHE.CLIENT_IP is reserved for users. However things became more complex with syslog-ng OSE 3.2. Let's see what sources generate name-value pairs: - traditional macros (e.g. $DATE); these are not name-value pairs per-se, but behave much like them, except that they are read-only - syslog message fields (e.g. $MSG) if the message is coming from a syslog source - filters whenever the 'store-matches' flag is set and the regexp contains groups - rewrite rules, whenever the rewrite rule specifies a thus far unknown name-value pair, e.g. set("something" value("name-value.pair")); - and of course parsers when you tell syslog-ng to parse an input as a CSV, or use db-parser together with the patterns produced by the patterndb projectThe latest stuff generating name-value pairs is the support for process accounting logs, in this case even the syslog related fields are missing and only things like "pacct.ac_comm" (to contain the program name) are defined. So I was thinking whether it should be "pacct.ac_comm" or ".pacct.ac_comm". With the quoted rule it should be simple: it is generated by syslog-ng itself, thus it should be in the syslog-ng namespace and should start with a dot. However in the era of syslog-ng plugins, what consists of syslog-ng at all? First, I wanted to use "pacct.ac_comm" (e.g. without a dot), because I liked this name better. I was trying to explain myself why it would not violate the rule above. The explanation I had for myself was: I'm going to "register" names such as this in the patterndb SCHEMAS.txt file. With this - not yet published - explanation, I've committed a patch to convert the pacctformat plugin to use a dotless prefix. Next, I was figuring that it is true that process accounting creates name-value pairs without going through patternization, but I've felt, that nothing ensures that these name-value pairs would be directly usable, when trying to analyse the logs. The patterndb concept uses tags and schemas to convert the incoming unstructured data into a consistent structure. However, pacct may not completely match what the user needs. And, in the future, when SNMP traps or SQL table polling are going to be supported, it is going to be even more true: these name-value pairs may need a conversion: from the SNMP/pacct structure to the patterndb schema described structure in order to handle these message sources consistently with regular syslog (and to make it easy to correllate these). So at the end, I've committed another patch, this time going back to ".pacct" as a prefix and leaving the original naming rule intact. The "pacct" prefix is up to the users to use, they may want the same information in a "pacct" schema, but that may come from data not directly tied from process accounting (e.g. from syslog messages). So this post is about doing nothing with regards to the naming policy, but I thought it'd be important to shed a light behind the scenes. Giving such decisions enough thought and coming up a with a long-term plan makes our lives much easier in the future. This post may be a bit more involved than the others, but feel free to ask me to elaborate, if you are interested. -- Posted By Bazsi to Bazsi's blog at 8/06/2010 08:26:00 PM

1 0

greedy parser still broken?
by Jiann-Ming Su 06 Aug '10

06 Aug '10

I've reopened bug 87 (https://bugzilla.balabit.com/show_bug.cgi?id=87) because I think the greedy flag is still broken. I've updated the bug with my observations. Can someone else offer any more insight? Thanks.

2 4

Re: [syslog-ng] Strange behavior with the option "mark"
by Ilas, Yann 06 Aug '10

06 Aug '10

Thank you for your answer. When I start the syslog-ng with the option "mark" sets to 0, no MARKS messages are sent (as written in the OSE 3.1 Administrator Guide). This problem appears only when the option "mark" is set from a value <> 0 to a value = 0 and when I reload the configuration with a HUP signal. Regards, Yann I. On Tue, 2010-08-03 at 10:58 +0200, Ilas, Yann wrote: > Hello everybody, > > I'm currently using the application syslog-ng version 3.1.1 and there is > an odd behavior with the "mark" option. > > Here is the configuration file used for the test : > > @version: 3.0 > > options { > mark(7); > }; > > source s_local { internal(); unix-stream( "/dev/log" ); }; > > destination d_f_msg_unknown { > file ( /var/log/messages__unknown ); > }; > > log { > source (s_local); > destination (d_f_msg_unknown); > }; > > ...and I start the syslog-ng application like this : > # cd /opt/syslog-ng/sbin/ > # ./syslog-ng -Fevd --foreground --no-caps > --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK > --pidfile=/tmp/syslog-ng.conf.pid > > The output of the log file : > Aug 2 17:44:22 serveur01 -- MARK -- > Aug 2 17:44:29 serveur01 -- MARK -- > Aug 2 17:44:36 serveur01 -- MARK -- > (...) > > Then, I change the configuration file from "mark(7)" to "mark(0)" and > send a HUP signal between "17:44:36" and "17:44:43" > # kill -HUP $(cat /tmp/syslog-ng.conf.pid) > > Here is the output (tail -f /var/log/messages__unknown) : > Aug 2 17:44:36 serveur01 -- MARK -- > <HUP signal> > Aug 2 17:44:43 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > (...) > ...then the server syslog-ng sends a lot of "MARK" messages. I have to > stop the process with a "kill" or "ctrl+c". > > I have the same behaviour if I use "mark_freq" instead of "mark". > What's wrong with that option ? Did I miss something ? well, we should recognize mark(0) the same as mark(-1) effectively disabling the mark feature. right now, mark(0) means that there's zero time between two mark messages, effectively generating one mark message per poll iteration, this is what you see. Or, does this happen only if you do a SIGHUP? Or the same happens when you start syslog-ng? -- Bazsi

3 3

2 syslogng processes for the price of 1
by Manson, John 05 Aug '10

05 Aug '10

Good Day, I start syslog-ng, ver3.0.4, from /usr/local/sbin using the same script I run with older versions of ng on other servers but this is the first time I have seem 2 processes started. Ng is loading the correct conf file at /usr/local/etc/syslog-ng.conf and is writing log messages to the correct log files, no duplicate entries. Always 2 pids in sequence. If I kill the smaller pid, 3532 in this case, the ppid of 1 takes over for 3532 and ng, 3533, runs along just fine. Help? Thanks root 3532 1 0 15:09:54 pts/2 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid root 3533 3532 0 15:09:54 ? 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid John Manson CAO/HIR/NI/Data-Communications | U.S. House of Representatives | Washington, DC 20515 Desk: 202-226-4244 | Team: 202-225-5552 | john.manson(a)mail.house.gov

3 2