[syslog-ng] PatternDB: macros extraction from URLs

[Gergely Nagy]

> We are using OSE 3.2.1 version and till now we have managed to
> configure most of the patterns we need.
> 
> However we have reached to a point where we need some hints from the
> users of this list. The problem is the following: how to extract
> macros when the order of them are not known (like in a URL).
> 
> For example we would like to extract 'user' and 'action' from an URL
> like the one mentioned below:
> 
> APP[9988]: WEB[0011]:
> http://abc.example.com/query.php?user=test1&action=login&host=prod1&device=device1
> HTTP 1.1
> 
> Unfortunately the "user" and "action" could be placed anywhere in the
> URL (as the URL is not created by aour application) thus we have to
> create something like this:
> 
> <pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&amp;@action=@ESTRING:action:&amp;@</pattern>
> <pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&amp;@ESTRING::a@ction=@ESTRING:action:&amp;@</pattern>
> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&amp;@user=@ESTRING:user:&amp;@</pattern>
> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&amp;@ESTRING::u@ser=@ESTRING:user:&amp;@</pattern>
> and so on....

While replying to Robert, I had another idea... I'm not sure how
feasible it would be, since I've never done anything similar, and my
knowledge in this area is almost nonexistent.

But!

URL parameters could be extracted as a single string first, and fed to a
CSV-parser, that'd do the job of extracting the fields. You'd need to
combine patterndb with the CSV parser, though, and I'm not quite sure
how one would go about doing that, or if it's even possible.

But it's worth a shot.

If that fails, another option would be to pre-process the logs, via a
short - say - perl script, that parses the URLs and rearranges the
parameters into a specific order, and adds empty values for any missing
parameters, so you'll only need a single pattern later. Then first send
the logs to a program() destination where the script does its stuff, and
then deliver that output back into syslog-ng, at which point it can be
easily processed with patterndb.

-- 
|8]