[syslog-ng] Struggling with regexp, store-matches and csv-parser macros

[Balazs Scheidler]

On Fri, 2011-08-05 at 14:08 +0200, Remy Zandwijk wrote:
> Hi list,
> 
> so, I am new to syslog-ng and I am struggling with the features
> regexp/store-matches and csv-parser macros. I searched the web and
> mailinglist archive for these two subjects, but that didn't answer my
> questions. The syslog-ng 3.1.3 I'm using is running on Debian Squeeze.
> 
> filter f_mytest {
>     facility(local0) and
>     level(debug) and
>     match("^www" value("MSGHDR") flags("store-matches"));
> };
> 
> template t_mytest {
>     template("${MSG}--${1}\n");
> }
> 
> destination d_mytest {
>     file("/var/tmp/mytest.log");
> };
> 
> log {
>     source(s_src);
>     filter(f_mytest);
>     template(t_mytest);
>     destination(f_mytest);
> };
> 
> 
> When I send a message to syslog-ng with: logger -p local0.debug -t www
> testmessage
> nothing appears in the logfile. However, when flags("store-matches")
> is omitted, the message appears like expected. Why does this filter
> not work?

hmm... that's strange. that flag shouldn't affect whether the filter
matches, it only specifies whether the regexp groups (the things in the
parentheses) should be stored into the message, which is costly and in
the majority of cases are not needed.

I've tried to test with your configuration, and I've found numerous
issues:

  * the template reference should be used in your file destination and
not on the log path, so you destination should look like this:

destination d_mytest {
    file("/var/tmp/mytest.log" template(t_mytest));
};

  * this also means that the template() option in your log statement
should be removed.

  * also, if you are only playing, you don't have to define your
template at the top-level, you can do it like this:


destination d_mytest {
    file("/var/tmp/mytest.log" template("${MSG}--${1}\n"));
};

  * the destination name had a typo, you defined d_mytest, and you tried
to reference f_mytest, this will be reported as a syntax error.

After that, syslog-ng starts with the configuration, but you've
triggered a bug, that causes syslog-ng to crash. More details on the
crash below, but a workaround is to match against the "PROGRAM" value
instead of MSGHDR.

With the crash fixed, it expands to nothing in ${1} since the regexp you
were using doesn't contain any groups. ${0} does (as that contain the
whole matched string), fixing the regexp causes syslog-ng this line to
be written into your destination file:

testmessage--www

And here's the configuration as I was using it:

@version: 3.1

source s_src { unix-stream("log"); };

filter f_mytest {
    facility(local0) and
    level(debug) and
    match("^www" value("MSGHDR") flags("store-matches"));
};

template t_mytest {
    template("${MSG}--${0}\n");
};

destination d_mytest {
    file("/var/tmp/mytest.log" template(t_mytest));
};

log {
    source(s_src);
    filter(f_mytest);
    destination(d_mytest);
};


Crash analysis:
===============

The root cause of the crash is that syslog-ng tries to be very smart
when storing name-value pairs into a log message: when it can, it only
stores an "indirect" value, which references another name-value pair and
an offset + length.

E.g. if you match against the $MESSAGE nvpair, and you match 10
characters at offset 10, then instead of storing a fully-blown value, it
just stores a reference to $MESSAGE and the offset (10) and length (10).

The issue is that MSGHDR that you were using is not an actual name-value
pair stored in the message, but rather, it is a value derived from other
fields, and is provided for usability reasons.

But anyway, here's a patch that fixes this problem for me in 3.3, it can
quite probably backported easily to older versions.

> 
> I am playing around with the csv-parser function as well. The
> (user-defined) macros can be used in file() functions, but I found out
> that they don't work in owner() and group() functions. Is that
> expected?
> 
> Thanks,
> Remy
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 

-- 
Bazsi