[syslog-ng] consecutive pattern parsers, and some other pattern matching questions

[Matthew Hall]

On Thu, Feb 10, 2011 at 01:54:52PM +0100, Valentijn Sessink wrote:
> Hello list,
> 
> I'm trying to setup a pattern DB for Postfix, and I'm running into the a
> couple of problems.
> 
> 1) One of the log messages I would like to catch is:
>  connect from smtp.example.com[xxxx:xxxx:300:40c1::23]
> Now I'd like to catch both hostname and IP-address, and maybe I'd like
> to feed the IP address into some sort of program later. So I thought I'd
> better put these in individual variables. However, the matching rule
> <pattern>connect from
> @ESTRING:postfix.remotehost:[@@IPvANY:postfix.remoteip@]</pattern>
> 
> doesn't work, because of the double "@@" - which is handled as an
> escaped "@", instead of two consecutive pattern parsers.
> 
> So my first question is: how can I have two consecutive pattern parsers
> in a pattern?

One possible workaround. Capture it all together.

Then make a rewrite rule to break it into two vars, when the 
.classifier.id matches the id you have for this rule.

> 2) Other messages say things like:
> 64A7F3001E7: from=<something at example.com> .....
> The syslog-ng OSE admin guide tells me to use @QSTRING:<>@ to match the
> mail address; but this shows an error. Is @QSTRING:&lt;&gt@  the correct
> way to proceed? (Or is this impossible with the current implementation?)

That's correct... XML escape the characters.

If you can run the output through W3C XML Tidy utility that helps hugely 
to make sure everything is right, and nicely indented to be readable.

http://packages.debian.org/sid/tidy

(Also available in many distros, I even use it in OS X Ports tree)

> 3) My third question boils down to: is it possible to correlate *one*
> single message into *two* separate trails?

I don't see why it wouldn't be possible if you had a syslog-ng source 
listening to mail.* and feeding through a single parser. Correlation is 
only available in syslog-ng 3.2 and up and I'm not doing correlation yet 
because I'm doing that work with higher level language code.

> 4) and finally: is there a good way to immediately end a certain
> context-scope? For example, after "disconnect from
> smtp.example.com[xxxx:xxxx:300:40c1::23]", the smtp phase is over, so
> there's no use keeping this context in memory anymore. Would adding
> something like "context-timeout=0" to the "disconnect" pattern work?

Probably is a way but I'm not sure what it is. Maybe one of the others 
that's done the SNG correlation could help.

> Best regards,
> Valentijn

HTH,
Matthew.