[syslog-ng] consecutive pattern parsers, and some other pattern matching questions

[Balazs Scheidler]

On Sun, 2011-02-20 at 17:00 +0100, Valentijn Sessink wrote:
> Op 20-02-11 14:25, Balazs Scheidler schreef:
> >> Yes, you can, but at a cost. To match one message with two patterns, you
> >> will need two different pattern databases:
> >> parser db1 {db_parser(file("/var/lib/syslog-ng/db1.xml"));};
> >> parser db2 {db_parser(file("/var/lib/syslog-ng/db2.xml"));};
> > Can you explain why you needed this? Why couldn't you do all processing
> > in your single rule?
> 
> My question came from Postfix, where i tried correlating the smtpd 
> "connect" and "disconnect" messages - which is quite trivial; but also 
> would like a larger correlation that included the whole mail delivery.
> 
> The connect/disconnect trail is simple: context-id="postfix-smtpd" 
> context-scope="process" and off you go.
> 
> The mail delivery trail is trickier: you cannot get the full trail with 
> just a "process" scope, you need to look for the "queueid". This queueid 
> starts with smtpd, so there you go: a single message from smtpd that has 
> a meaning in two different contexts.
> 
> Please note that the queue-id is not available in all smtpd messages, so 
> it is not possible to add trail 1 to trail 2.
> 
> (I hope my explanation is clear, if not, please say so; I have a couple 
> of patterns and also a postfix log trail that I could include).
> 

That really is a problem, you basically need two correllation states for
the same message, while I originally envisioned one. In fact the first
designs permitted this scenario as well, but the final design doesn't.

Do you use the same pattern in this case? e.g. are your name-value pairs
the same in the two rules?

If this is the case, then this could be supported by simply associating
two rules with the same pattern (which is internally a separate
'object').

Something like this:

<rule id="1" context-id='foo' context-scope='process'>
  <pattern>postfix pattern</pattern>
</rule>
<rule id="2" context-id='foo.$queue_id' context-scope='host'>
  <pattern>postfix pattern</pattern>
</rule>

Although this would cause some problems, because syslog-ng currently
assumes that each message matches a rule or it doesn't. It currently
doesn't have the notion of multiple matches.

Also, I'm not sure this would be very intuitive. Currently we display an
error message in pattern collisions.

Marci, what do you think?

-- 
Bazsi