[syslog-ng] 3.7.1 patterndb matching rule preference fix - incorrect

[Evan Rempel]

That does not really address the issue. By adding @ANYSTRNG@ to then 
end, we just end up with
equally long matches, and then the pattern database load order needs to 
be controlled to define
preference. When merging pattern databases there is no real control over 
load order. Additionally,
if you and an @ANYSTRING@ to the end of a pattern that is designed to 
match the whole line, you
fail to get a match at all because @ANYSTRING@ can not match a zero 
length string.

The real issue is that the most specific pattern needs to be the 
preferred pattern when
there are multiple matches. The discussion is how to define a pattern as 
being more specific.

Consider glob file patterns as an example. Listed from least to most 
specific,
or stated differently, matching the most number of files, to matching 
the least number of files.

*
*.doc
contract*.doc*
contract*2015*.doc
contract-purchase-2015.doc

The reason that this list is least specific to most specific is because 
there is more
static content to match on.

If these were mail filtering rules to store e-mail in different folders, 
you would never expect
the document "contract-sale-2015.doc" to be stored in a folder 
associated with the rule for
the pattern "contract*.doc*". You would expect the rule 
"contract*2015*.doc" to have priority.
It is the amount of static content in the pattern that defines how 
specific a pattern is.

In the above example, it looks easy because the longest pattern is the 
most specific. That is
misleading because using Syslog-ng pattern syntax, a user variable name 
can be introduced
which can make a longer pattern actauly match a shorter message. For example

some @STRING:my.variable.matched:"@ here

compared to

some @STRING@ here @STRING@ done

The second is more specific.

At my site we had already done this because we store all of our patterns 
in a database and
programmatically create our pattern database by ordering the patterns by 
the amount of static
content. Now that the patterndb was "fixed" in 3.7 we can't use this 
work around any more, which
makes 3.7 break our pattern database :-(

Evan.


On 09/25/2015 02:35 PM, Scheidler, Balázs wrote:
>
> What if we grabbed the tail of the message you are matching with 
> @ANYSTRING@ to a name-value pair automatically, so you don't need 
> anything in your rule, making it a shorter match than the other.
>
> What do you think?
>
> On Sep 22, 2015 10:43 PM, "Fabien Wernli" <wernli at in2p3.fr 
> <mailto:wernli at in2p3.fr>> wrote:
>
>     Hi Evan,
>
>     On Tue, Sep 22, 2015 at 09:49:43AM -0700, Evan Rempel wrote:
>     > I propose that the PatternDB preference be changed from the
>     pattern with the longest MATCH to the pattern with the largest
>     amount of static content.
>
>     I fully agree with Evan here: it should work as described in this
>     sentence.
>     That being said, I'm not so sure about the Status quo with 3.7.1.
>     Maybe Balázs can give some more details on the change?
>
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> This body part will be downloaded on demand.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150925/8eb18411/attachment.htm