[syslog-ng] PatternDB: macros extraction from URLs

[Fekete Robert]

Hi,

I have an idea how this could be accomplished, but it is strictly speculative 
and untested, and also requires OSE 3.3, because you'll have to use mongodb.

First, you'll have to separate your variables into generic macros using parsers, 
since you do not know their order. For example, from 
user=test1&action=login&host=prod1&device=device1 you could make $myvariable1 
and $myvalue1, $myvariable2 and $myvalue2, and so on, where the value of 
$myvariable1 is user, the value of $myvalue1 is test1, etc.

Next, you put your messages to mongodb. The trick here is that you do not 
explicitly know the name of the fields you have for each message, but it doesn't 
matter because mongodb will handle that (i _think_ so, but Algernon will 
probably correct me if it doesn't). So when setting the field names for the 
mongodb driver, you'll use macros, something like:

fields("$myvariable1", "$myvariable2");
values("$myvalue1", "$myvalue2");
(or whatever the syntax of the driver is)

Does this sound reasonable?

Robert

On 02/17/2011 09:31 AM, Ioan Indreias wrote:

> Hello all,
>
> We are using OSE 3.2.1 version and till now we have managed to
> configure most of the patterns we need.
>
> However we have reached to a point where we need some hints from the
> users of this list. The problem is the following: how to extract
> macros when the order of them are not known (like in a URL).
>
> For example we would like to extract 'user' and 'action' from an URL
> like the one mentioned below:
>
> APP[9988]: WEB[0011]:
> http://abc.example.com/query.php?user=test1&action=login&host=prod1&device=device1
> HTTP 1.1
>
> Unfortunately the "user" and "action" could be placed anywhere in the
> URL (as the URL is not created by aour application) thus we have to
> create something like this:
>
> <pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&amp;@action=@ESTRING:action:&amp;@</pattern>
> <pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&amp;@ESTRING::a@ction=@ESTRING:action:&amp;@</pattern>
> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&amp;@user=@ESTRING:user:&amp;@</pattern>
> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&amp;@ESTRING::u@ser=@ESTRING:user:&amp;@</pattern>
> and so on....
>
> Not to mention if we need to extract the 'device' macro as well - the
> number of patterns grow significantly.
>
> Have somebody some hints on how to optimize the extraction of macros
> when them are not in an known order?
>
> Thank you in advance,
> Ioan
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>