On Fri, 2011-08-05 at 14:08 +0200, Remy Zandwijk wrote:

Hi list,

so, I am new to syslog-ng and I am struggling with the features regexp/store-matches and csv-parser macros. I searched the web and mailinglist archive for these two subjects, but that didn't answer my questions. The syslog-ng 3.1.3 I'm using is running on Debian Squeeze.

filter f_mytest { facility(local0) and level(debug) and match("^www" value("MSGHDR") flags("store-matches")); };

template t_mytest { template("${MSG}--${1}\n"); }

destination d_mytest { file("/var/tmp/mytest.log"); };

log { source(s_src); filter(f_mytest); template(t_mytest); destination(f_mytest); };

When I send a message to syslog-ng with: logger -p local0.debug -t www testmessage nothing appears in the logfile. However, when flags("store-matches") is omitted, the message appears like expected. Why does this filter not work?

hmm... that's strange. that flag shouldn't affect whether the filter matches, it only specifies whether the regexp groups (the things in the parentheses) should be stored into the message, which is costly and in the majority of cases are not needed. I've tried to test with your configuration, and I've found numerous issues: * the template reference should be used in your file destination and not on the log path, so you destination should look like this: destination d_mytest { file("/var/tmp/mytest.log" template(t_mytest)); }; * this also means that the template() option in your log statement should be removed. * also, if you are only playing, you don't have to define your template at the top-level, you can do it like this: destination d_mytest { file("/var/tmp/mytest.log" template("${MSG}--${1}\n")); }; * the destination name had a typo, you defined d_mytest, and you tried to reference f_mytest, this will be reported as a syntax error. After that, syslog-ng starts with the configuration, but you've triggered a bug, that causes syslog-ng to crash. More details on the crash below, but a workaround is to match against the "PROGRAM" value instead of MSGHDR. With the crash fixed, it expands to nothing in ${1} since the regexp you were using doesn't contain any groups. ${0} does (as that contain the whole matched string), fixing the regexp causes syslog-ng this line to be written into your destination file: testmessage--www And here's the configuration as I was using it: @version: 3.1 source s_src { unix-stream("log"); }; filter f_mytest { facility(local0) and level(debug) and match("^www" value("MSGHDR") flags("store-matches")); }; template t_mytest { template("${MSG}--${0}\n"); }; destination d_mytest { file("/var/tmp/mytest.log" template(t_mytest)); }; log { source(s_src); filter(f_mytest); destination(d_mytest); }; Crash analysis: =============== The root cause of the crash is that syslog-ng tries to be very smart when storing name-value pairs into a log message: when it can, it only stores an "indirect" value, which references another name-value pair and an offset + length. E.g. if you match against the $MESSAGE nvpair, and you match 10 characters at offset 10, then instead of storing a fully-blown value, it just stores a reference to $MESSAGE and the offset (10) and length (10). The issue is that MSGHDR that you were using is not an actual name-value pair stored in the message, but rather, it is a value derived from other fields, and is provided for usability reasons. But anyway, here's a patch that fixes this problem for me in 3.3, it can quite probably backported easily to older versions.

I am playing around with the csv-parser function as well. The (user-defined) macros can be used in file() functions, but I found out that they don't work in owner() and group() functions. Is that expected?

Thanks, Remy ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-- Bazsi