August 2010 - syslog-ng - lists.balabit.hu

[Bazsi's blog] syslog-ng name-value pair naming
by Bazsi 06 Aug '10

06 Aug '10

I was giving a lot of thought recently to the topic of naming name-value pairs in syslog-ng. Until now the only documented rule is stating somewhat vaguely that whenever you use a parser you should choose a name that has at least one dot in it, and this dot must not be the initial character. This means that names like MSG or .SDATA.meta.sequenceId are reserved for syslog-ng, and APACHE.CLIENT_IP is reserved for users. However things became more complex with syslog-ng OSE 3.2. Let's see what sources generate name-value pairs: - traditional macros (e.g. $DATE); these are not name-value pairs per-se, but behave much like them, except that they are read-only - syslog message fields (e.g. $MSG) if the message is coming from a syslog source - filters whenever the 'store-matches' flag is set and the regexp contains groups - rewrite rules, whenever the rewrite rule specifies a thus far unknown name-value pair, e.g. set("something" value("name-value.pair")); - and of course parsers when you tell syslog-ng to parse an input as a CSV, or use db-parser together with the patterns produced by the patterndb projectThe latest stuff generating name-value pairs is the support for process accounting logs, in this case even the syslog related fields are missing and only things like "pacct.ac_comm" (to contain the program name) are defined. So I was thinking whether it should be "pacct.ac_comm" or ".pacct.ac_comm". With the quoted rule it should be simple: it is generated by syslog-ng itself, thus it should be in the syslog-ng namespace and should start with a dot. However in the era of syslog-ng plugins, what consists of syslog-ng at all? First, I wanted to use "pacct.ac_comm" (e.g. without a dot), because I liked this name better. I was trying to explain myself why it would not violate the rule above. The explanation I had for myself was: I'm going to "register" names such as this in the patterndb SCHEMAS.txt file. With this - not yet published - explanation, I've committed a patch to convert the pacctformat plugin to use a dotless prefix. Next, I was figuring that it is true that process accounting creates name-value pairs without going through patternization, but I've felt, that nothing ensures that these name-value pairs would be directly usable, when trying to analyse the logs. The patterndb concept uses tags and schemas to convert the incoming unstructured data into a consistent structure. However, pacct may not completely match what the user needs. And, in the future, when SNMP traps or SQL table polling are going to be supported, it is going to be even more true: these name-value pairs may need a conversion: from the SNMP/pacct structure to the patterndb schema described structure in order to handle these message sources consistently with regular syslog (and to make it easy to correllate these). So at the end, I've committed another patch, this time going back to ".pacct" as a prefix and leaving the original naming rule intact. The "pacct" prefix is up to the users to use, they may want the same information in a "pacct" schema, but that may come from data not directly tied from process accounting (e.g. from syslog messages). So this post is about doing nothing with regards to the naming policy, but I thought it'd be important to shed a light behind the scenes. Giving such decisions enough thought and coming up a with a long-term plan makes our lives much easier in the future. This post may be a bit more involved than the others, but feel free to ask me to elaborate, if you are interested. -- Posted By Bazsi to Bazsi's blog at 8/06/2010 08:26:00 PM

1 0

greedy parser still broken?
by Jiann-Ming Su 06 Aug '10

06 Aug '10

I've reopened bug 87 (https://bugzilla.balabit.com/show_bug.cgi?id=87) because I think the greedy flag is still broken. I've updated the bug with my observations. Can someone else offer any more insight? Thanks.

2 4

Re: [syslog-ng] Strange behavior with the option "mark"
by Ilas, Yann 06 Aug '10

06 Aug '10

Thank you for your answer. When I start the syslog-ng with the option "mark" sets to 0, no MARKS messages are sent (as written in the OSE 3.1 Administrator Guide). This problem appears only when the option "mark" is set from a value <> 0 to a value = 0 and when I reload the configuration with a HUP signal. Regards, Yann I. On Tue, 2010-08-03 at 10:58 +0200, Ilas, Yann wrote: > Hello everybody, > > I'm currently using the application syslog-ng version 3.1.1 and there is > an odd behavior with the "mark" option. > > Here is the configuration file used for the test : > > @version: 3.0 > > options { > mark(7); > }; > > source s_local { internal(); unix-stream( "/dev/log" ); }; > > destination d_f_msg_unknown { > file ( /var/log/messages__unknown ); > }; > > log { > source (s_local); > destination (d_f_msg_unknown); > }; > > ...and I start the syslog-ng application like this : > # cd /opt/syslog-ng/sbin/ > # ./syslog-ng -Fevd --foreground --no-caps > --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK > --pidfile=/tmp/syslog-ng.conf.pid > > The output of the log file : > Aug 2 17:44:22 serveur01 -- MARK -- > Aug 2 17:44:29 serveur01 -- MARK -- > Aug 2 17:44:36 serveur01 -- MARK -- > (...) > > Then, I change the configuration file from "mark(7)" to "mark(0)" and > send a HUP signal between "17:44:36" and "17:44:43" > # kill -HUP $(cat /tmp/syslog-ng.conf.pid) > > Here is the output (tail -f /var/log/messages__unknown) : > Aug 2 17:44:36 serveur01 -- MARK -- > <HUP signal> > Aug 2 17:44:43 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > Aug 2 17:44:50 serveur01 -- MARK -- > (...) > ...then the server syslog-ng sends a lot of "MARK" messages. I have to > stop the process with a "kill" or "ctrl+c". > > I have the same behaviour if I use "mark_freq" instead of "mark". > What's wrong with that option ? Did I miss something ? well, we should recognize mark(0) the same as mark(-1) effectively disabling the mark feature. right now, mark(0) means that there's zero time between two mark messages, effectively generating one mark message per poll iteration, this is what you see. Or, does this happen only if you do a SIGHUP? Or the same happens when you start syslog-ng? -- Bazsi

3 3

2 syslogng processes for the price of 1
by Manson, John 05 Aug '10

05 Aug '10

Good Day, I start syslog-ng, ver3.0.4, from /usr/local/sbin using the same script I run with older versions of ng on other servers but this is the first time I have seem 2 processes started. Ng is loading the correct conf file at /usr/local/etc/syslog-ng.conf and is writing log messages to the correct log files, no duplicate entries. Always 2 pids in sequence. If I kill the smaller pid, 3532 in this case, the ppid of 1 takes over for 3532 and ng, 3533, runs along just fine. Help? Thanks root 3532 1 0 15:09:54 pts/2 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid root 3533 3532 0 15:09:54 ? 0:00 /usr/local/sbin/syslog-ng -p /usr/local/etc/syslog-ng/syslog-ng.pid John Manson CAO/HIR/NI/Data-Communications | U.S. House of Representatives | Washington, DC 20515 Desk: 202-226-4244 | Team: 202-225-5552 | john.manson(a)mail.house.gov

3 2

Debugging Pattern Match Failures
by Matthew Hall 04 Aug '10

04 Aug '10

Hello list, Recently I created a series of blasphemous scripts which convert some large collections of recorded log messages in my environment into pattern DB XML files. At first there were some syntax errors but I fixed all of these and the XML files are loading successfully. However I am running into some problems with the next step: getting the patterns to match against the incoming log messages. I suspect I am not properly stripping the headers off of the disk files of recorded messages I am using to generate the pattern DB XML files. I am wondering how I can enable the right debugging capabilities to get more detailed debug output from the pattern DB parser where I can see what strings are being processed so that I can fix this right instead of guessing repeatedly and incorrectly. Thanks, Matthew Hall.

3 12

syslog-ng 3.1.2 has been released
by devel＠balabit.hu 04 Aug '10

04 Aug '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.1.2 SUMMARY : new stable release DATE : Aug 4, 2010 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.1.2) has been released. For latest fixes in the 3.1.x feature branch you are recommended to upgrade to this version. CHANGES: 3.1.2 Mon, 02 Aug 2010 17:06:28 +0200 INCOMPATIBLE CHANGES: * This release changes the default for 'store-legacy-msghdr' flag for log sources. In previous releases this had to be enabled explicitly, because of the performance penalties, which do not apply to 3.1. The original behaviour can be restored by explicitly specifying 'dont-store-legacy-msghdr'. See this blog post for more details: http://bazsi.blogs.balabit.com/2010/05/small-incompatible-change-for-31.html Bugfixes: * When syslog-ng is reloaded, the local hostname value was not refreshed, causing syslog-ng to remember the hostname until the next restart. This may not play nice with DHCP configured hostnames, which may change dynamically. * When several SIGHUP signals are received in quick succession, the last one may have been dropped. This problem is fixed. * Make sure that numbers never get resolved using getpwnam/getgrnam (e.g. libnss functions), because this may cause deadlocks if the NSS provider is LDAP and the LDAP server is trying to log a message to syslog about invalid usernames. * Fixed flush_timeout() handling in for non-file destinations. Enabling flush_lines()/flush_timeout() for a non-file destination, either using a global option, or a per-destination option could cause excess CPU usage as long as the destination wasn't writable. * In case patterndb has a syntax error when reloading the file, syslog-ng automatically reverts to the old version instead of dropping pattern matching altogether. * Fixed pdbtool to properly handle accented characters in the command line. * Fixed the use of the greedy and drop-invalid flags for csv-parser. * Fixed a possible but rare memory leak in the handling of message suppression. Other changes: * Added "update-patterndb" command that merges patterndb files from ${sysconfdir}/patterndb.d and produces ${localstatedir}/patterndb.xml, the default file of the db-parser(). * Make it possible to specify the path to the syslog-ng control socket using a command line option in order to make it easier to launch multiple syslog-ng instances. * Introduced a new, cleaner syntax to specify that the permissions of a file are not to be changed. Use owner/group/perm options of a file without any parameters instead of "-1". The old mechanism had the problem that both per-file and global settings had to use "-1" to effectively stop syslog-ng from changing the uid/gid/perm values. The new syntax will make it possible to specify defaults in the global options section and customize those on a per-file basis. * syslog-ng will not go into the background if either the --syntax-only or --debug options are specified. Loggen: * Added SSL support. * Added the ability to read from a sample file Credits: * We would like to send our thank for all contributors sending us bugreports, documentation issues or patches. They are: * Marius Tomaschewski * Corey Hickey * Evan Rempel * Andy Kitchingman * andrewn at locus.net Hopefully noone was missed from this list. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Strange behavior with the option "mark"
by Ilas, Yann 03 Aug '10

03 Aug '10

Hello everybody, I'm currently using the application syslog-ng version 3.1.1 and there is an odd behavior with the "mark" option. Here is the configuration file used for the test : @version: 3.0 options { mark(7); }; source s_local { internal(); unix-stream( "/dev/log" ); }; destination d_f_msg_unknown { file ( /var/log/messages__unknown ); }; log { source (s_local); destination (d_f_msg_unknown); }; ...and I start the syslog-ng application like this : # cd /opt/syslog-ng/sbin/ # ./syslog-ng -Fevd --foreground --no-caps --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK --pidfile=/tmp/syslog-ng.conf.pid The output of the log file : Aug 2 17:44:22 serveur01 -- MARK -- Aug 2 17:44:29 serveur01 -- MARK -- Aug 2 17:44:36 serveur01 -- MARK -- (...) Then, I change the configuration file from "mark(7)" to "mark(0)" and send a HUP signal between "17:44:36" and "17:44:43" # kill -HUP $(cat /tmp/syslog-ng.conf.pid) Here is the output (tail -f /var/log/messages__unknown) : Aug 2 17:44:36 serveur01 -- MARK -- <HUP signal> Aug 2 17:44:43 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- (...) ...then the server syslog-ng sends a lot of "MARK" messages. I have to stop the process with a "kill" or "ctrl+c". I have the same behaviour if I use "mark_freq" instead of "mark". What's wrong with that option ? Did I miss something ? Regards, Yann I.

2 1

[Bug 84] New: Program destination blocks if program stops reading input
by bugzilla＠bugzilla.balabit.com 02 Aug '10

02 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=84 Summary: Program destination blocks if program stops reading input Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Syslog-ng OSE 3.0.5 and 3.0.6 - binary download from balabit and self compiled all behave the same. If the program in a destination stops reading its input, syslog-ng buffers a small number of messages and then stops reading the source (in my case a pipe). I would expect syslog-ng to read from its input until the log_fifo_size limit was reached and then drop messages. an strace is attached. The configuration file is -------------- options { log_fifo_size(1000000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; template t_application { template("$MESSAGE\n"); template_escape(no); }; template t_localsyslog { template("<$PRI>$S_ISODATE $HOST erempel: $MSGONLY\n"); template_escape(no); }; source local { pipe("/var/log/syslog.pipes/erempel" flags(no-parse) ); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) template(t_localsyslog) log_fifo_size(5000) ); }; destination erempel { program("/home1l/erempel/read-noread.pl" template(t_application) log_fifo_size(8000000) ); }; log { source(local); destination(erempel); }; log { source(int); destination(localsyslog); }; ---------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8