[syslog-ng] dbparser problem isolating messages

[Balint Kovacs]

Hi Evan,

I would say, this is partially intended. DBParser will alway do prefix 
matching and this is a good thing, as you do not need to describe the 
whole message. This way it is less prone to message format changes, if 
anything is appended to the end of the message, the pattern will still 
match and many times only the leading portion of the message needs to be 
parsed.

I'm actively using this assumption in my patterns and others are 
probably doing the same, so changing this behaviour would result in many 
patterns not matching anymore. Probably a parser marking the end of 
message would be a good solution. That's a bit out of the logic of 
patterndb but if memory serves me right it won't be too hard to do. A 
possible (completely untested) workaround would be to use the PCRE 
parser in 3.4 HEAD and use $ to match the end of string.

I'm not sure why the order matters, others knowing the dbparser codebase 
better probably have an answer off the top of their heads. I would 
assume that dbparser should always do a best match and not linear 
evaluation, so I'd say this is not intended and is worth a bugreport.

The above is also valid for the program pattern, however, you can use 
parsers in that pattern as well, and the SET parser should be more 
performant than a regex.

Balint

On 10/31/2012 05:01 PM, Evan Rempel wrote:
> I am having a problem with the dbparser when messages have identical leading portions.
>
> In my case I have two messages
>
> xlog: backup pg_xlog/000000010000014700000076
> xlog: backup pg_xlog/000000010000014700000076 failed
>
>
> The first of these two messages is a success, and should be safely ignored.
> The second of these two messages is a failure, and I want to tag it and route it to a ticket creation program
> so that our Operations Center can investigate.
>
> Patterns are;
>
> xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@
> xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed
>
> If these patterns are placed into the pattern database in this order, then the first pattern will match the failed log
> message, which is completely wrong. I would ignore the message rather than route it to my ticketing system.
>
> If I change the order of them, then all works correctly.
>
> I don't think that "hand tuning" the xml file is the correct approach, and in my case can not actually be done.
> My patterndb.xml file is created by a program that reads all of the patterns and tests messages from an external database.
>
> I've worked around this by sorting the patterns descending by the length of the patterns, but if I had the two patterns;
>
> xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@
> xlog: backup pg_xlog/@ESTRING:: @failed
>
> then that "trick" does not work either.
>
> As I see it, the dbparser should match complete messages only. That would make the order completely irrelevant.
>
>
>
>
>
>
>
> Incidentally, I have the same problem with the "program" matching that the dbparser users.
>
> Take the mimedefang program as an example. There are multiple components with program names of;
>
> mimedefang
> mimedefang-multiplexor
> mimedefang.pl
>
> but the program mimedefang will match them all. This means that the pattern database file must either;
>
> 1. Have all of the message patterns under mimedefang
> 2. have the program sections sorted in reverse order so that the shortest program matches last.
>
> Again, I think that the pattern database should match the complete program name. Using multiple patterns for the program
> name is technically sufficient, but it would be nice to use a regular expression so that I can match
>
> imapd
> imapds
>
> with the program
> impads?
>
>
> As usual, all comments welcome, especially those that explain the history of why it was done this way and
> how the issue can be addressed.
>
>
> Thanks to everyone for such an active community.
>
> Evan.
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>