We are using OSE 3.2.1 version and till now we have managed to configure most of the patterns we need.
However we have reached to a point where we need some hints from the users of this list. The problem is the following: how to extract macros when the order of them are not known (like in a URL).
For example we would like to extract 'user' and 'action' from an URL like the one mentioned below:
APP[9988]: WEB[0011]: http://abc.example.com/query.php?user=test1&action=login&host=prod1&device=d... HTTP 1.1
Unfortunately the "user" and "action" could be placed anywhere in the URL (as the URL is not created by aour application) thus we have to create something like this:
<pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&@action=@ESTRING:action:&@</pattern> <pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&@ESTRING::a@ction=@ESTRING:action:&@</pattern> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&@user=@ESTRING:user:&@</pattern> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&@ESTRING::u@ser=@ESTRING:user:&@</pattern> and so on....
While replying to Robert, I had another idea... I'm not sure how feasible it would be, since I've never done anything similar, and my knowledge in this area is almost nonexistent. But! URL parameters could be extracted as a single string first, and fed to a CSV-parser, that'd do the job of extracting the fields. You'd need to combine patterndb with the CSV parser, though, and I'm not quite sure how one would go about doing that, or if it's even possible. But it's worth a shot. If that fails, another option would be to pre-process the logs, via a short - say - perl script, that parses the URLs and rearranges the parameters into a specific order, and adds empty values for any missing parameters, so you'll only need a single pattern later. Then first send the logs to a program() destination where the script does its stuff, and then deliver that output back into syslog-ng, at which point it can be easily processed with patterndb. -- |8]