[syslog-ng] Thoughts on patterndb syntax

[Balazs Scheidler]

Hi Lars,

First of all thanks for your message. Any kind of feedback is very much
appreciated, basically these make me want to work on the code further :)

So I'd like to urge everyone to post their opinions, they really make my
day.

Comments on your points are below, inserted into your message.

On Wed, 2010-10-20 at 21:57 -0400, Lars Kellogg-Stedman wrote:
> I've been playing with 3.2beta1 recently and getting my feet wet with
> the patterndb support, which I haven't had a chance to work with
> before.  I have a few thoughts regarding the patterndb rule syntax,
> mostly targeted at making things a little bit easier to work with.
> 
> - Rule IDs
> 
> Is there any particular reason why unique IDs were selected as rule
> identifiers?  They're not particularly meaningful to people, and
> they're hard to talk about.  It's much easier to say, "we're suddently
> seeing lots of matches for ssh-accept-connection" than it is to say,
> "we're suddenly seeing lots of matches for
> 4dd5a329-da83-4876-a431-ddcb59c2858c".  With the current version of
> syslog-ng it looks like it's possible to use arbitrary identifiers in
> place of UUIDs, and that's what I'm doing for my local rulesets.
> 
> This even makes classification metadata more useful, because
> .classifier.rule_id=ssh-accept-connection is immediately meaningful,
> while a UUID is useless unless I go grepping around the database.

Well, I don't really made up my mind how rule_id's should be used. They
were proposed by Marci (who implemented the patterndb syntax in the
first place). syslog-ng doesn't really care, but they must be unique.

These are useful as we can attach a lot of information to the patterndb
rule. For example, if you use a "<description>" tag, then you can
retrieve this said description when you browse the log, based on the
unique ID.

This is what SSB (our syslog-ng appliance product) does for example.



> 
> - Whitespace
> 
> Log messages tend to be long, which makes them unwieldy in a number of
> situations.  It would be nice if instead of this:
> 
>   <pattern>...some very long log message that makes my life difficult
> and my patterns hard to read ...</pattern>
> 
> I could do this:
> 
>   <pattern collapse_whitespace="yes">...some very long log message
>   that has been conveniently wrapped to
>   that it's easier to edit, email, and otherwise
>   work with.</pattern>
> 
> Specifically, enabling "collapse_whitespace" would transform any
> sequence of whitespace to a single " ".
> 
> As proposed here this would be a completely backwords-compatible
> change because the default behavior would remain the same.

Interesting idea and of course doable, but then if there's indeed
multiple spaces in the message, you get in trouble.

> 
> - Reusable patterns
> 
> We use Cisco firewall modules on our network.  When these devices log
> a connection-related message, the source and destination address look
> something like this:
> 
>   ircs:3610:67.186.94.126/41004
> 
> That's:
> 
>   interface:vlan:ip/port
> 
> Which becomes:
> 
>   @ESTRING:fwsm.src_if::@@NUMBER:fwsm.src_vlan@:@IPv4:fwsm.src_ip@/@NUMBER:fwsm.src_port@
> 
> When this occurs throughout the ruleset, and multiple times within a
> single message, it really lowers the readability of the rules.  I wish
> there was a way to modularize this so that I could create custom
> types, something like this:
> 
>  <type name="FWSM_ADDRSPEC">
>  @ESTRING:iface::@@NUMBER:vlan@:@IPv4:ip@/@NUMBER:fwsm.port@
>  </type>
> 
> And then do this:
> 
>   <pattern>Accepted src @FWSM_ADDRSPEC:fwsm.src:@ dst
> @FWSM_ADDRSPEC:fwsm.dst:@</pattern>
> 
> And get this:
> 
>   fwsm.src.iface
>   fwsm.src.ip
>   fwsm.src.port

This is a good idea. Although I'm a bit fiddling with the idea to extend
the patterndb syntax a little bit. Nothing concrete yet, but reusable
components will necessarily be a part of the picture.

-- 
Bazsi