[syslog-ng] pdbtool 'patternize'

[David Hauck]

Hi Péter,
 
On Wednesday, April 16, 2014 10:11 AM, syslog-ng-bounces at lists.balabit.hu wrote:
> Hi,
> 
> On Wed, Apr 16, 2014 at 6:15 PM, David Hauck <davidh at netacquire.com>
> wrote:
> 	Another couple questions regarding 'patternize'.
> 
> 	Why does the 'patternize' output not include additionally relevant 
> parts of the schema? In particular the 'program pattern' is not output 
> as part of the result? It's my understanding that this is key matching 
> criteria when determining matches and I'm unsure what would happen 
> with the pattern db that contains rulesets with no program pattern 
> specifiers (note: the documentation does talk about the matching 
> behaviour when ${PROGRAM} is empty, but this is different - i.e., I 
> assume rules with empty program patterns don't get matched/looked at when ${PROGRAM} is non-empty).
> 
> That's because the clustering algorithm used within patternize itself 
> does not take the program field into account, so including that in the 
> pattern database would create erroneous results. It wouldn't be that 
> difficult to update the algorithm to use the program field and only 
> group logs together if they have the same value there but I won't have 
> time to get to it in the upcoming weeks. It's a low hanging fruit if 
> you are willing to code, I am happy to help if you get stuck :)
> 
> If the {$PROGRAM} is non-empty but there's no "program" entry defined 
> in the pattern, the message does get matched, although I am pretty 
> sure that the patterns where the "program" entry is specified are 
> stronger, but I am not 100% about that priority order. Actually, 
> that's what happens if you run "pdbtool test" on an XML generated by
> patternize: as you can see it contains examples in which the program 
> field is set to the bogus "patternize" value manually, and the 
> patterns match those examples nevertheless. Probably the documentation should be updated to describe that scenario, too.

OK, I get the gist of all of the above and so my remaining question is then: "what's the point of the 'program pattern' in the ruleset definitions"? 

> 	Also, where is the actual schema (the xsd file) that defines the 
> pattern db format (and the semantics of each element)? I've found the 
> admin guide documentation lacking in terms of explicit description of 
> the patter db format (the brief section that attempts to describe this 
> is very thin).
> 
> Well, a human-readable description can indeed never be as precise as a 
> formal definition :) I don't know how the version you are using is 
> packaged, but in the source tree these XSDs are in "/doc/xsd":
> https://github.com/balabit/syslog-ng/tree/master/doc/xsd These are 
> pretty well annotated XSDs which should be quite self-explaining when 
> it comes to the semantics, too.

Great, thx - I'll take a look (maybe it will help to clarify my remaining question above ;)).

Thanks for this,
-David
 
> greets,
> Peter