- syslog-ng - lists.balabit.hu

[Bazsi's blog] LWN: syslog-ng rotten to the (Open) Core?
by Bazsi 08 Aug '10

08 Aug '10

This was first posted as a comment under an article on lwn.net, but I thought it was important enough to post it here for others not reading lwn. Please go ahead and read the original article which is about the "Open Core" business model and its problems from the Free Software community point of view. A commenter thought that syslog-ng was an example, which only exists as a marketing tool for the company's commercial offering. Anyway, here's my post: First of all, I want to make it clear that I'm biased on the syslog-ng case, but still wanted to express my opinion here. I'm biased as I'm the primary author of syslog-ng. I think syslog-ng is a completely different case from the one described by Neary. The GPL version is not crippleware, it was never published for marketing purposes only and for the majority of syslog-ng's existence only the Open Source stuff existed. The Premium Edition is only about 3 years old and syslog-ng started in 1998. We never removed features from the OSE version, the Premium Edition only included _additional_ features, and a lot of those are already available in the OSE. Some examples: * TLS support (became available in 3.0, almost 2 years ago) * SQL destination (became available in 2.1, 2.5 years ago) * performance improvements (3.0) * etc. In the other direction, we usually receive bugfixes and it is a pure technical reason that we used to require copyright assignment: I wanted to keep the two branches as close as possible (which if not done is the reason #1 why Open Core products become crippleware fast). _And_ since we heavily invested in automatic testing and our customers report bugs directly to us, we fix way more bugs in the OSE version than the community. But anyway, I didn't think that the dual license model was so problematic at the time we made this decision 3 years ago. Our efforts have never been "Rotten to the Open Core". If you don't believe that, check out the git repository or read the mailing list archive and see it yourself. And this whole mess is the past, OSE 3.2 has been relicensed, and it is true that we're going to publish non-free plugins, but anyone else is welcome to join and do the same. -- Posted By Bazsi to Bazsi's blog at 8/08/2010 01:23:00 PM

1 0

[Bazsi's blog] syslog-ng 3.2alpha2 released
by Bazsi 07 Aug '10

07 Aug '10

I've just uploaded syslog-ng 3.2alpha2 to the release directory. The last alpha release didn't compile on all supported platforms and the automatic test-suite was disabled, because it only worked if syslog-ng got installed first. These obstacles have been overcome and together with some fixes and a couple of new features, 3.2alpha2 is now available. I've also forward ported all bugfixes from syslog-ng 3.1.2. For those who are starting to experiment with the 3.2 branch, here's the list of new features compared to 3.1. Those who tried 3.2alpha1, the list of changes compared to 3.2alpha1 is at the end of this post. Since the documentation of syslog-ng is not yet up-to-date with the new features introduced, I've tried to also include URLs for the best known descriptions. The references may not be 100% accurate, but should give anyone interested an idea how to start experimenting. Also, please note that although this is an alpha release, the bulk of the changes are in the configuration parser, so once your configuration was parsed properly and syslog-ng starts up, an almost unchanged code is processing it. This means that this release should be good enough to start playing with. And feedback about what kind of syslog-ng.conf parsing errors you encounter on real-life configuration files is more than welcome. Code quality & functionality wise, this could be a beta release, I only expect "procedural" changes, like cleaning up the plugin names, which wouldn't be nice to do in a beta release (though not unheard of :) New features in 3.2: - Plugins: the new architecture replaces the old monolithic one, all syslog-ng functionality is loaded from external plugins when needed. It is possible to write plugins to extend syslog-ng functionality in the following areas: sources, destinations, filter expression, parsers, rewrite ops, message format. http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.ht… - The framework for a "syslog-ng configuration library" (aka SCL) a collection of configuration snippets installed along syslog-ng, simplifying the authoring of syslog-ng configuration files.http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=287993339599deac0442e26355c600b5aee63583 http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.ht… - pdbtool match is now able to read a file containing syslog messages and apply patterndb and a filter expression on the contents.http://bazsi.blogs.balabit.com/2010/07/patterndb-grep-on-steroids.… - pdbtool test is now able to perform pattern testing automatically based on the supplied example log message.http://marci.blogs.balabit.com/2010/07/pdbtool-test-and-pattern-dat… - Persistent state containing the current file position for file sources is now continously updated during runtime, instead of updating it only at exit, which makes it much more reliable in case syslog-ng doesn't terminate normally. - Better syntax error reporting in the configuration file.http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html - Support for reusable configuration snippets, similar to macros with parameters, named "blocks". http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-opened-experimental-blo… - Added a confgen plugin that includes the output of a program into the configuration file, making it possible to generate configuration file snippets dynamically. http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=5248ef6c49ff3af… - Support for BSD-style process accounting logs via the pacct() source driver defined in by SCL and the underlying pacctformat plugin.http://bazsi.blogs.balabit.com/2010/07/syslog-ng-and-process-account… - Support for explicit COMMITs in the SQL driver, this speeds up SQL INSERT rate significantly if flush_lines() is non-zero. http://bazsi.blogs.balabit.com/2010/04/explicit-transaction-support-in-sql.… - It is now possible to supply a filter to rewrite expressions and only apply the rewrite rule in case the filter matches.https://lists.balabit.hu/pipermail/syslog-ng/2010-July/014565.html - It is now possible to use multiple parser expressions in a single parser object, similar to rewrite rules. - Added support for using the include statement from anywhere in the configuration file, instead of only at top-level. Also introduced syslog-ng "global values" that can be defined and the substituted anywhere in the configuration file. http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=1203267c465256c… http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=52098762f27cde0… - Default configuration file supplied as part of SCL. Incompatible changes: - syslog-ng traditionally expected an optional hostname field even when a syslog message is received on a local transport (e.g. /dev/log). However no UNIX version is known to include this field. This caused problems when the application creating the log message has a space in its program name field. This behaviour has been changed for the unix-stream/unix-dgram/pipe drivers if the config version is 3.2 and can be restored by using an explicit 'expect-hostname' flag for the specific source. Changes since 3.2alpha1: - Now compiles on all platforms and the unit/functional tests also run. (tested: AIX, HP-UX, Solaris, FreeBSD, Linux, Tru64) - Fixed pdbtool match --debug-pattern output for ESTRING parsers. - Fixed a possible memory leak in the lexer, which would accumulate in case SIGHUPs. - Fixed Solaris STREAMS device support. - Forward ported all bugfixes from syslog-ng OSE 3.0 & 3.1 - Disable process accounting module by default as it doesn't compile on non-Linux platforms. - Added "pdbtool match --file" option to read and parse an existing logfile. - Added "pdbtool test" to check the log samples in the patterndb file. - Added "dont-create-tables" flag for the SQL destination to inhibit automatic table creation. - Added "condition()" support for rewrite expressions, which makes it possible to skip rewrite rules that do not match a filter expression. - Added "--module-path" command line option to control where modules are loaded from from the command line. Happy logging! -- Posted By Bazsi to Bazsi's blog at 8/07/2010 05:58:00 PM

1 0

[Bazsi's blog] syslog-ng name-value pair naming
by Bazsi 06 Aug '10

06 Aug '10

I was giving a lot of thought recently to the topic of naming name-value pairs in syslog-ng. Until now the only documented rule is stating somewhat vaguely that whenever you use a parser you should choose a name that has at least one dot in it, and this dot must not be the initial character. This means that names like MSG or .SDATA.meta.sequenceId are reserved for syslog-ng, and APACHE.CLIENT_IP is reserved for users. However things became more complex with syslog-ng OSE 3.2. Let's see what sources generate name-value pairs: - traditional macros (e.g. $DATE); these are not name-value pairs per-se, but behave much like them, except that they are read-only - syslog message fields (e.g. $MSG) if the message is coming from a syslog source - filters whenever the 'store-matches' flag is set and the regexp contains groups - rewrite rules, whenever the rewrite rule specifies a thus far unknown name-value pair, e.g. set("something" value("name-value.pair")); - and of course parsers when you tell syslog-ng to parse an input as a CSV, or use db-parser together with the patterns produced by the patterndb projectThe latest stuff generating name-value pairs is the support for process accounting logs, in this case even the syslog related fields are missing and only things like "pacct.ac_comm" (to contain the program name) are defined. So I was thinking whether it should be "pacct.ac_comm" or ".pacct.ac_comm". With the quoted rule it should be simple: it is generated by syslog-ng itself, thus it should be in the syslog-ng namespace and should start with a dot. However in the era of syslog-ng plugins, what consists of syslog-ng at all? First, I wanted to use "pacct.ac_comm" (e.g. without a dot), because I liked this name better. I was trying to explain myself why it would not violate the rule above. The explanation I had for myself was: I'm going to "register" names such as this in the patterndb SCHEMAS.txt file. With this - not yet published - explanation, I've committed a patch to convert the pacctformat plugin to use a dotless prefix. Next, I was figuring that it is true that process accounting creates name-value pairs without going through patternization, but I've felt, that nothing ensures that these name-value pairs would be directly usable, when trying to analyse the logs. The patterndb concept uses tags and schemas to convert the incoming unstructured data into a consistent structure. However, pacct may not completely match what the user needs. And, in the future, when SNMP traps or SQL table polling are going to be supported, it is going to be even more true: these name-value pairs may need a conversion: from the SNMP/pacct structure to the patterndb schema described structure in order to handle these message sources consistently with regular syslog (and to make it easy to correllate these). So at the end, I've committed another patch, this time going back to ".pacct" as a prefix and leaving the original naming rule intact. The "pacct" prefix is up to the users to use, they may want the same information in a "pacct" schema, but that may come from data not directly tied from process accounting (e.g. from syslog messages). So this post is about doing nothing with regards to the naming policy, but I thought it'd be important to shed a light behind the scenes. Giving such decisions enough thought and coming up a with a long-term plan makes our lives much easier in the future. This post may be a bit more involved than the others, but feel free to ask me to elaborate, if you are interested. -- Posted By Bazsi to Bazsi's blog at 8/06/2010 08:26:00 PM

1 0

greedy parser still broken?
by Jiann-Ming Su 06 Aug '10

06 Aug '10

I've reopened bug 87 (https://bugzilla.balabit.com/show_bug.cgi?id=87) because I think the greedy flag is still broken. I've updated the bug with my observations. Can someone else offer any more insight? Thanks.

2 4

Re: [syslog-ng] Strange behavior with the option "mark"
by Ilas, Yann 06 Aug '10

06 Aug '10

Thank you for your answer. When I start the syslog-ng with the option "mark" sets to 0, no MARKS messages are sent (as written in the OSE 3.1 Administrator Guide). This problem appears only when the option "mark" is set from a value <> 0 to a value = 0 and when I reload the configuration with a HUP signal. Regards, Yann I. On Tue, 2010-08-03 at 10:58 +0200, Ilas, Yann wrote: > Hello everybody, > > I'm currently using the application syslog-ng version 3.1.1 and there is > an odd behavior with the "mark" option. > > Here is the configuration file used for the test : > > @version: 3.0 > > options { > mark(7); > }; > > source s_local { internal(); unix-stream( "/dev/log" ); }; > > destination d_f_msg_unknown { > file ( /var/log/messages__unknown ); > }; > > log { > source (s_local); > destination (d_f_msg_unknown); > }; > > ...and I start the syslog-ng application like this : > # cd /opt/syslog-ng/sbin/ > # ./syslog-ng -Fevd --foreground --no-caps > --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK > --pidfile=/tmp/syslog-ng.conf.pid > > The output of the log file : > Aug 2 17:44:22 serveur01 -- MARK -- > Aug 2 17:44:29 serveur01 -- MARK -- > Aug 2 17:44:36 serveur01 -- MARK -- > (...) > > Then, I change the configuration file from "mark(7)" to "mark(0)" and > send a HUP signal between "17:44:36" and "17:44:43" > # kill -HUP $(cat /tmp/syslog-ng.conf.pid) > > Here is the output (tail -f /var/log/messages__unknown) : > Aug 2 17:44:36 serveur01 -- MARK -- > <HUP signal> > Aug 2 17:44:43 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > (...) > ...then the server syslog-ng sends a lot of "MARK" messages. I have to > stop the process with a "kill" or "ctrl+c". > > I have the same behaviour if I use "mark_freq" instead of "mark". > What's wrong with that option ? Did I miss something ? well, we should recognize mark(0) the same as mark(-1) effectively disabling the mark feature. right now, mark(0) means that there's zero time between two mark messages, effectively generating one mark message per poll iteration, this is what you see. Or, does this happen only if you do a SIGHUP? Or the same happens when you start syslog-ng? -- Bazsi

3 3

2 syslogng processes for the price of 1
by Manson, John 05 Aug '10

05 Aug '10

Good Day, I start syslog-ng, ver3.0.4, from /usr/local/sbin using the same script I run with older versions of ng on other servers but this is the first time I have seem 2 processes started. Ng is loading the correct conf file at /usr/local/etc/syslog-ng.conf and is writing log messages to the correct log files, no duplicate entries. Always 2 pids in sequence. If I kill the smaller pid, 3532 in this case, the ppid of 1 takes over for 3532 and ng, 3533, runs along just fine. Help? Thanks root 3532 1 0 15:09:54 pts/2 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid root 3533 3532 0 15:09:54 ? 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid John Manson CAO/HIR/NI/Data-Communications | U.S. House of Representatives | Washington, DC 20515 Desk: 202-226-4244 | Team: 202-225-5552 | john.manson(a)mail.house.gov

3 2

Debugging Pattern Match Failures
by Matthew Hall 04 Aug '10

04 Aug '10

Hello list, Recently I created a series of blasphemous scripts which convert some large collections of recorded log messages in my environment into pattern DB XML files. At first there were some syntax errors but I fixed all of these and the XML files are loading successfully. However I am running into some problems with the next step: getting the patterns to match against the incoming log messages. I suspect I am not properly stripping the headers off of the disk files of recorded messages I am using to generate the pattern DB XML files. I am wondering how I can enable the right debugging capabilities to get more detailed debug output from the pattern DB parser where I can see what strings are being processed so that I can fix this right instead of guessing repeatedly and incorrectly. Thanks, Matthew Hall.

3 12

syslog-ng 3.1.2 has been released
by devel＠balabit.hu 04 Aug '10

04 Aug '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.1.2 SUMMARY : new stable release DATE : Aug 4, 2010 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.1.2) has been released. For latest fixes in the 3.1.x feature branch you are recommended to upgrade to this version. CHANGES: 3.1.2 Mon, 02 Aug 2010 17:06:28 +0200 INCOMPATIBLE CHANGES: * This release changes the default for 'store-legacy-msghdr' flag for log sources. In previous releases this had to be enabled explicitly, because of the performance penalties, which do not apply to 3.1. The original behaviour can be restored by explicitly specifying 'dont-store-legacy-msghdr'. See this blog post for more details: http://bazsi.blogs.balabit.com/2010/05/small-incompatible-change-for-31.html Bugfixes: * When syslog-ng is reloaded, the local hostname value was not refreshed, causing syslog-ng to remember the hostname until the next restart. This may not play nice with DHCP configured hostnames, which may change dynamically. * When several SIGHUP signals are received in quick succession, the last one may have been dropped. This problem is fixed. * Make sure that numbers never get resolved using getpwnam/getgrnam (e.g. libnss functions), because this may cause deadlocks if the NSS provider is LDAP and the LDAP server is trying to log a message to syslog about invalid usernames. * Fixed flush_timeout() handling in for non-file destinations. Enabling flush_lines()/flush_timeout() for a non-file destination, either using a global option, or a per-destination option could cause excess CPU usage as long as the destination wasn't writable. * In case patterndb has a syntax error when reloading the file, syslog-ng automatically reverts to the old version instead of dropping pattern matching altogether. * Fixed pdbtool to properly handle accented characters in the command line. * Fixed the use of the greedy and drop-invalid flags for csv-parser. * Fixed a possible but rare memory leak in the handling of message suppression. Other changes: * Added "update-patterndb" command that merges patterndb files from ${sysconfdir}/patterndb.d and produces ${localstatedir}/patterndb.xml, the default file of the db-parser(). * Make it possible to specify the path to the syslog-ng control socket using a command line option in order to make it easier to launch multiple syslog-ng instances. * Introduced a new, cleaner syntax to specify that the permissions of a file are not to be changed. Use owner/group/perm options of a file without any parameters instead of "-1". The old mechanism had the problem that both per-file and global settings had to use "-1" to effectively stop syslog-ng from changing the uid/gid/perm values. The new syntax will make it possible to specify defaults in the global options section and customize those on a per-file basis. * syslog-ng will not go into the background if either the --syntax-only or --debug options are specified. Loggen: * Added SSL support. * Added the ability to read from a sample file Credits: * We would like to send our thank for all contributors sending us bugreports, documentation issues or patches. They are: * Marius Tomaschewski * Corey Hickey * Evan Rempel * Andy Kitchingman * andrewn at locus.net Hopefully noone was missed from this list. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Strange behavior with the option "mark"
by Ilas, Yann 03 Aug '10

03 Aug '10

Hello everybody, I'm currently using the application syslog-ng version 3.1.1 and there is an odd behavior with the "mark" option. Here is the configuration file used for the test : @version: 3.0 options { mark(7); }; source s_local { internal(); unix-stream( "/dev/log" ); }; destination d_f_msg_unknown { file ( /var/log/messages__unknown ); }; log { source (s_local); destination (d_f_msg_unknown); }; ...and I start the syslog-ng application like this : # cd /opt/syslog-ng/sbin/ # ./syslog-ng -Fevd --foreground --no-caps --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK --pidfile=/tmp/syslog-ng.conf.pid The output of the log file : Aug 2 17:44:22 serveur01 -- MARK -- Aug 2 17:44:29 serveur01 -- MARK -- Aug 2 17:44:36 serveur01 -- MARK -- (...) Then, I change the configuration file from "mark(7)" to "mark(0)" and send a HUP signal between "17:44:36" and "17:44:43" # kill -HUP $(cat /tmp/syslog-ng.conf.pid) Here is the output (tail -f /var/log/messages__unknown) : Aug 2 17:44:36 serveur01 -- MARK -- <HUP signal> Aug 2 17:44:43 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- (...) ...then the server syslog-ng sends a lot of "MARK" messages. I have to stop the process with a "kill" or "ctrl+c". I have the same behaviour if I use "mark_freq" instead of "mark". What's wrong with that option ? Did I miss something ? Regards, Yann I.

2 1

[Bug 84] New: Program destination blocks if program stops reading input
by bugzilla＠bugzilla.balabit.com 02 Aug '10

02 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=84 Summary: Program destination blocks if program stops reading input Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Syslog-ng OSE 3.0.5 and 3.0.6 - binary download from balabit and self compiled all behave the same. If the program in a destination stops reading its input, syslog-ng buffers a small number of messages and then stops reading the source (in my case a pipe). I would expect syslog-ng to read from its input until the log_fifo_size limit was reached and then drop messages. an strace is attached. The configuration file is -------------- options { log_fifo_size(1000000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; template t_application { template("$MESSAGE\n"); template_escape(no); }; template t_localsyslog { template("<$PRI>$S_ISODATE $HOST erempel: $MSGONLY\n"); template_escape(no); }; source local { pipe("/var/log/syslog.pipes/erempel" flags(no-parse) ); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) template(t_localsyslog) log_fifo_size(5000) ); }; destination erempel { program("/home1l/erempel/read-noread.pl" template(t_application) log_fifo_size(8000000) ); }; log { source(local); destination(erempel); }; log { source(int); destination(localsyslog); }; ---------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

Pattern for Eventlog-to-Syslog
by Martin Holste 31 Jul '10

31 Jul '10

I've recently started using the great util Eventlog-to-Syslog (http://code.google.com/p/eventlog-to-syslog/) and needed a parser. This pattern should do the job. A word of warning, I opted to go with no program match because I didn't want to write a pattern per event log source, so it's possible this would hit on other log sources. However, in testing, the pattern seems to work well and I haven't gotten any "false" matches. I also gave up trying to parse out the user name as the second param because it isn't always included and the semicolon used as a delimiter regularly shows up naturally in messages. One could write per-event-id message parsers which grabbed all kinds of fields, but this should get the ball rolling to at least grab the event id. <ruleset name="Windows" id='4'>  <rules> <rule provider="local" class='Windows' id='4'> <patterns> <pattern>@NUMBER:event_id:@: @ANYSTRING:msg:@</pattern> </patterns> <examples> <example> <test_message program="Service_Control_Manager">7035: NT AUTHORITY\SYSTEM: The COH_Mon service was successfully sent a start control.</test_message> <test_values> <test_value name="event_id">7035</test_value> <test_value name="msg">NT AUTHORITY\SYSTEM: The COH_Mon service was successfully sent a start control.</test_value> </test_values> </example> <example> <test_message program="SceCli">1202: Security policies were propagated with warning. 0x4b8 : An extended error has occurred. For best results in resolving this event, log on with a non-administrative account and search http://support.microsoft.com for "Troubleshooting Event 1202's".</test_message> <test_value name="event_id">1202</test_value> <test_value name="msg">Security policies were propagated with warning. 0x4b8 : An extended error has occurred. For best results in resolving this event, log on with a non-administrative account and search http://support.microsoft.com for "Troubleshooting Event 1202's".</test_value> </example> </examples> <tags> <tag>Windows</tag> </tags> </rule> </rules> </ruleset>

2 2

Syntax in 3.0 Admin Guide not working for MySQL
by Chuck 31 Jul '10

31 Jul '10

I am trying to log to mysql. I am running syslog-ng 3.0.4 on Solaris 10 Update 8. I have the following stanza in syslog-ng.conf: destination r_mysql { sql(type(mysql) host("localhost") username("syslog") password("xxxxxxx") database("syslog") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime", "host", "program", "pid", "message") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; It is complaining about the syntax.. Anyone have any ideas? Thx

4 4

Syslog IO Error
by Pedro Rafael Alves Simoes 30 Jul '10

30 Jul '10

Hello, My Syslog-ng shows this error message: I/O error occurred while reading; fd='183', error='Connection reset by peer (104)' Anybody have a clue in how to resolve this? I've tried a google search about this but didn't get much information... Using RedHat Linux 2.6.18-164.el5 syslog-ng-2.1.4-1.el5 Thanks.

2 1

Solaris 10 Compilation Woes
by Chuck 30 Jul '10

30 Jul '10

Trying to build 3.1.1 on Solaris 10 (update 8) so I can enable SQL support. I've installed the required version of gcc, bison, flex, and etc.. per the INSTALL doc. Here is the error I get: (I've searched google all morning and find tons of others with this error but no solution) ./configure --prefix=/usr/local --enable-sql --enable-pcre checking for pkg-config... /bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for a BSD-compatible install... /usr/local/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /usr/local/bin/mkdir -p checking for gawk... no checking for mawk... no checking for nawk... nawk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking for gcc... (cached) gcc checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed checking dependency style of gcc... (cached) gcc3 checking for gcc option to accept ISO C99... -std=gnu99 checking for bison... bison -y checking for flex... flex checking lex output file root... configure: error: cannot find output from flex; giving up I am using the following environment vars: CXX=g++ CFLAGS=-I/usr/local/include CPPFLAGS=-I/usr/local/include CC=gcc All my open source tools are under /usr/local. I have tried have /usr/ccs/bin in my PATH before /yusr/local/bin, tried having it after, and tried not having it in my PATH at all. Anyone have any ideas? Thx, CC

2 3

Re: [syslog-ng] Small problem with pdbtool and the parser ?
by Ilas, Yann 29 Jul '10

29 Jul '10

Hello, Thanks ! It works... Regards, Yann I. Hi, Please use this patch (fix by Viktor Tusa): diff --git a/src/dbparser/pdbtool.c b/src/dbparser/pdbtool.c index 5ce3e79..cde859a 100644 --- a/src/dbparser/pdbtool.c +++ b/src/dbparser/pdbtool.c @@ -11,6 +11,7 @@ #include <stdio.h> #include <string.h> #include <stdlib.h> +#include <locale.h> #if HAVE_GETOPT_H #include <getopt.h> @@ -618,6 +620,7 @@ main(int argc, char *argv[]) usage(); } + setlocale(LC_ALL,""); if (!g_option_context_parse(ctx, &argc, &argv, &error)) { fprintf(stderr, "Error parsing command line arguments: %s\n", error ? error->message : "Invalid arguments"); M On Wed, 2010-07-28 at 15:39 +0200, Ilas, Yann wrote: > Hello, > > I have a small problem with pdbtool. When I try to match this message "> Message message n?2 message 255", I obtain this error : "Error parsing command line arguments: Invalid byte sequence in conversion input". > The problem seems to be the "?" which is bad interpreted by pdbtool. But the syslog-ng application has not that problem. > > Here is the xml file I used : > <?xml version='1.0' encoding='UTF-8'?> > <patterndb version='3' pub_date='2010-07-28'> > <ruleset name='test' id='ab-cd-123456789'> > <pattern>test</pattern> > <rules> > <rule provider='yann' id='xxxxx:syslog:abcdef:1234567890:id001' class='system'> > <patterns> > <pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern> > <values> > <value name=".classifier.facility">local5</value> > <value name=".classifier.severity">notice</value> > <value name=".classifier.priority">173</value> > </values> > </patterns> > </rule> > </rules> > </ruleset> > </patterndb> > > If I use the pattern "<pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern>", I obtain : > $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message n?2 message 255' > Error parsing command line arguments: Invalid byte sequence in conversion input > => It's not OK > > If I use the pattern "<pattern>> Message message nx@NUMBER@ message @NUMBER@</pattern>", I obtain : > (the diffence is the charactere "x" after the " n") > $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message nx2 message 255' > MESSAGE=> Message message nx2 message 255 > PROGRAM=test > .classifier.class=system > .classifier.rule_id=xxxxx:syslog:abcdef:1234567890:id001 > .classifier.facility=local5 > .classifier.severity=notice > .classifier.priority=173 > => It's OK > > With syslog-ng and the pattern "<pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern>", I tried to send the same message (with the "?") by using "nc" and it works, ie the message is well interpreted by the application : > $ LANG=C date +"<70>%b %d %R:%S server.domain test[1234]: > Message message n?123 message 456" | nc -4 127.0.0.1 514 > > Why pdbtool can't parse that string ? > > Regards, > > Yann I. >

1 0

[patterndb] more login/logout samples
by Balazs Scheidler 28 Jul '10

28 Jul '10

Hi, Just as I wrote in the other thread, I'm almost finished with marking up the log samples that you posted here on this list. This means that we can keep the ball rolling. I guess there are other services running on your computers doing authentication, so please go ahead and email bomb me with further samples. :) Also, I'd appreciate some review on the markup, and some testing with real life data. See my blog post about "grep on steroids" to see how you can immediately use the stuff finished so far. -- Bazsi

1 0

patterndb: collect login/logout samples
by Balazs Scheidler 28 Jul '10

28 Jul '10

Hi, After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone. My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication. As a starter, I've commited access/sshd.pdb, containing three rules for OpenSSH login/logout/login failure events. I'd head towards standard services, ftp, pop3 and imap authentication, using their "default" implementation in Ubuntu/Debian. (if there's no default, I'll just pick one at random). If any of you can collect these 3 samples of any of the applications that you run daily on your system and submit them here, it'd be tremendous use and would be appreciated. The format of the submission would be preferred in patterndb format (see the ssh sample I've just pushed), but if you are afraid of that, even simple samples would be useful, I'll do the markup myself. -- Bazsi

9 27

Small problem with pdbtool and the parser ?
by Ilas, Yann 28 Jul '10

28 Jul '10

Hello, I have a small problem with pdbtool. When I try to match this message "> Message message n°2 message 255", I obtain this error : "Error parsing command line arguments: Invalid byte sequence in conversion input". The problem seems to be the "°" which is bad interpreted by pdbtool. But the syslog-ng application has not that problem. Here is the xml file I used : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-28'> <ruleset name='test' id='ab-cd-123456789'> <pattern>test</pattern> <rules> <rule provider='yann' id='xxxxx:syslog:abcdef:1234567890:id001' class='system'> <patterns> <pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern> <values> <value name=".classifier.facility">local5</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">173</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> If I use the pattern "<pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern>", I obtain : $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message n°2 message 255' Error parsing command line arguments: Invalid byte sequence in conversion input => It's not OK If I use the pattern "<pattern>> Message message nx@NUMBER@ message @NUMBER@</pattern>", I obtain : (the diffence is the charactere "x" after the " n") $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message nx2 message 255' MESSAGE=> Message message nx2 message 255 PROGRAM=test .classifier.class=system .classifier.rule_id=xxxxx:syslog:abcdef:1234567890:id001 .classifier.facility=local5 .classifier.severity=notice .classifier.priority=173 => It's OK With syslog-ng and the pattern "<pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern>", I tried to send the same message (with the "°") by using "nc" and it works, ie the message is well interpreted by the application : $ LANG=C date +"<70>%b %d %R:%S server.domain test[1234]: > Message message n°123 message 456" | nc -4 127.0.0.1 514 Why pdbtool can't parse that string ? Regards, Yann I.

2 1

sql() and pgsql
by John Kristoff 28 Jul '10

28 Jul '10

Hello, First thanks to Bazsi and co. for years of great work on syslog-ng. I've used it off and on for some time and have always found it to be well implemented and supported. I've experimented a bit with rsyslog since some distros have been making it their new default and found syslog-ng to be easier to work with and more reliable for my tasks. I'm been playing around with v3.0.8 and the sql() / pgsql output driver to insert log data into a specific table on a specific schema. I couldn't figure out how to specify the schema and maybe its just not supoprted. For instance, if the following is defined: database: foo schema: bar table: baz but the PostgreSQL search path for the user doesn't include bar, then syslog-ng will attempt to create a table in the default schema (usually 'public'). I tried setting the table name to "bar.baz", but that just created a table named bar_baz in the default schema. I did a quick check in the source code and didn't see where this was occurring, maybe its a function of a third party library? If schema support was available that would be nice. For posterity sake and maybe a FAQ, the work around is to alter the search_path for the user (e.g. ALTER ROLE user SET search_path = bar,public;). On a related note, it might be nice to have an option to avoid creating a table if it does not already exist. If nothing else, this helps limit the permissions granted to the syslog-ng db user, which would be appreciated. Likewise, it might also be nice to have an option that can disable checking if the table exists (the "SELECT * FROM %s WHERE 0=1" query) to further lock down user permissions. John

3 3

config file parsing bugs in syslog-ng 3.0.7
by Matthew Hall 28 Jul '10

28 Jul '10

Hello list, In the course of my work I discovered two small defects in syslog-ng syslog-ng 3.0.7. Since I am not familiar enough with the code to patch these I am trying to do the best I can by reporting them to the experts. There are workarounds for these. I am in the process of doing some other work on the product which I would like to contribute back in the near future after I figure out how the open-source contribution process works at this new company I joined. Thank you for providing this product. I am impressed with the documentation and all of the capabilities and I look forward to becoming more expert with it over the coming few months. Best Regards, Matthew Hall. 1) specifying the flags directive in a log stanza other than in the last position causes a syntax error This specific format works: log { filter(filter_local1); destination(dst_local1); parser(parse_switch); flags(final); }; Any format like this raises an error: log { filter(filter_local1); destination(dst_local1); flags(final); parser(parse_switch); }; Is there some reason why or just a parser glitch? 2) using a version directive newer than the daemon version has unexpected behavior If I feed 3.0 a file marked with @version: 3.1 tag, I get this error: Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; I think it does not make sense to fall back to the oldest format if the file has a newer version tag. Would it be better to try to process the file as the newest version with a warning, or print an error that the file is too new for 3.0 to read?

2 1

Pipe() in syslog-ng ---- HELP
by Khaleelah Peerbocus 28 Jul '10

28 Jul '10

Hi How can I avoid line breaks in syslog-ng when it is configured using pipe destination d_mssql { pipe("/tmp/mssql.syslog-ng.pipe" flags(no-multi-line) template("INSERT INTO logs(datetime,host,program,pid,message)VALUES('$R_DATE','$HOST','$PROGRAM',' $PID','$MSGONLY');") template-escape(yes)); }; log { source(src); filter(f_debug); destination(d_mssql); }; Then I have a script reading the pipe if [ -e /tmp/mssql.syslog-ng.pipe ]; then while [ -e /tmp/mssql.syslog-ng.pipe ] do isql -v MSTEST "sa" "password" < /tmp/mssql.syslog-ng.pipe done else mkfifo /tmp/mssql.syslog-ng.pipe fi Problem is the sql query is getting broken down in 3 lines KP

2 12

prevent syslog-ng to collect syslogd messages
by Hoenig, Stefan, VF-Group 27 Jul '10

27 Jul '10

Hi Guys, we use syslog-ng to collect application logs and still use the standard syslogd for system logging. The application logs were defined in an include config file so the application teams are able to configure them as they need it. The problem is that we want to prevent that they can also configure systemlogging sources in their config like "/proc/kmsg" or "dev/log" on Linux. Is there a possibility to overwrite such sources in the global config or better a global parameter that avoids such configurations? Thanks in advance.... With best regards / Mit freundlichen Grüßen Stefan Hoenig

2 1

Question about QSTRING and "<>" separators
by Ilas, Yann 26 Jul '10

26 Jul '10

Hello, I have another question (the last of the day ?) about db parser. In fact, my question is about the QSTRING. I'm not sur to understand how to use it. I follow the example given here (http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog -ng-db.html) but it doesn't seem to work. Here is my xml file : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-23'> <ruleset name='test' id='yann-test'> <description>Programme : test</description> <pattern>test</pattern> <rules> <rule provider='yann' id='123456789' class='system'> <patterns> <pattern>@QSTRING:message:<>@ Message message message</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> I try to test a message by using the pdbtool : $ /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P test -M "<123> Message message message" Key contains '@' without escaping; key='@QSTRING:message:', value='123456789' Error parsing pattern database file; filename='/tmp/syslog-ng.xml', error='Error on line 9 char 38: \'>\' is not a valid character following a \'<\' character; it may not begin an element name' ...and I obtain this error. What's wrong with the message or the use of QSTRING ? I replaced '<>' by "'" (simple quote) and it works... Regards, Yann I.

2 1

Pattern database and '<'
by Ilas, Yann 26 Jul '10

26 Jul '10

Hello, Currently, I'm using syslog-ng version 3.1.1. I would like to parse that kind of message by using pattern database. Here is the message : "<<<<< Message message message". So I created that xml file : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-22'> <ruleset name='test' id='yann-test-9999999999'> <description>Programme : test</description> <pattern>test</pattern> <rules> <rule provider='yann' id='ss-test:syslog:dest:123123123:id005' class='system'> <patterns> <pattern><<<<< Message message message</pattern> <values> <value name=".classifier.facility">local0</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">133</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> I obtain that message when I try to parse with the pdbtool application : $ pdbtool match -p syslog-ng.xml -P test -M "<<<<< Message message message" Error parsing pattern database file; filename='/tmp/syslog-ng.xml', error='Error on line 9 char 22: \'<\' is not a valid character following a \'<\' character; it may not begin an element name' Into the xml file, I replaced '<<<<<' with @STRING::@ or @ESTRING...@ but it doesn't work. Is it possible to match that kind of message ? Regards, Yann

3 2

How to deal with duplicate log entries
by Chuck 26 Jul '10

26 Jul '10

I have the following log statgements.. (Im basically trying to mirror what solaris 10's default syslogl.conf does): log { source (s_streams); filter (f_emerg); destination (l_messages); }; log { source (s_streams); filter (f_err); destination (l_messages); }; log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; log { source (s_streams); filter (f_daemon); filter (f_notice); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_crit); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; # sshd logging However when sending to daemon.err I get duplicate messages... Should I change line 2 to this: log { source (s_streams); filter (f_err); filter (not f_daemon) destination (l_messages); }; Or should I hard-code every facility like so: log { source (s_streams); filter (f_err); filter (f_daemon) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_mail) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_auth) destination (l_messages); }; ... and so on? Thx,.CC

2 1