[syslog-ng] PatternDB: macros extraction from URLs

[Gergely Nagy]

On Thu, 2011-02-17 at 14:11 +0100, Fekete Robert wrote:
> I have an idea how this could be accomplished, but it is strictly speculative 
> and untested, and also requires OSE 3.3, because you'll have to use mongodb.

The method described by Robert _might_ work with 3.2 too, with other
destinations, but your mileage may vary, and it has a few shortcomings.

> First, you'll have to separate your variables into generic macros using parsers, 
> since you do not know their order. For example, from 
> user=test1&action=login&host=prod1&device=device1 you could make $myvariable1 
> and $myvalue1, $myvariable2 and $myvalue2, and so on, where the value of 
> $myvariable1 is user, the value of $myvalue1 is test1, etc.

As far as I understand, this'd mean that instead of having a pattern for
each and every combination of key=value pairs in the URLs, you'd only
have N patterns, where N is the maximum number of parameters you want to
support. Not the best, but better than what was explained in the
original mail.

In each pattern, you'd extract not "user=$value", but "$key1=$value1".
This has the downside that you wouldn't be able to explicitly address
the 'user' field, but if you only want to extract the field and log it
(and not filter on it), then that shouldn't be a problem: one can use
$key1=$value1 in templates.

Your patterns would look something like this:

<pattern>http://abc.example.com/query.php@ESTRING:key1:=@@ESTRING:value1:&amp;</pattern>

And so on... My patterndb knowledge is sadly lacking, but hopefully its
understandable what I'm trying to say :)

> Next, you put your messages to mongodb. The trick here is that you do not 
> explicitly know the name of the fields you have for each message, but it doesn't 
> matter because mongodb will handle that (i _think_ so, but Algernon will 
> probably correct me if it doesn't). So when setting the field names for the 
> mongodb driver, you'll use macros, something like:
>
> fields("$myvariable1", "$myvariable2");
> values("$myvalue1", "$myvalue2");

At the moment, that won't work, as the field names cannot be templates
in the mongodb driver yet (I thought it could be, but now that I looked
at the code, I was wrong).

However, 3.3 will have a solution for that, in the form of
value-pairs(). I won't go into detail about that just yet, since the
code is not ready yet.

For the time being, if you're ok with logging the parameters, and you
don't need to filter on them, then there's a possible solution, even for
3.2, for any destination, based on the suggestions Robert gave.

-- 
|8]