Hi Evan, I would say, this is partially intended. DBParser will alway do prefix matching and this is a good thing, as you do not need to describe the whole message. This way it is less prone to message format changes, if anything is appended to the end of the message, the pattern will still match and many times only the leading portion of the message needs to be parsed. I'm actively using this assumption in my patterns and others are probably doing the same, so changing this behaviour would result in many patterns not matching anymore. Probably a parser marking the end of message would be a good solution. That's a bit out of the logic of patterndb but if memory serves me right it won't be too hard to do. A possible (completely untested) workaround would be to use the PCRE parser in 3.4 HEAD and use $ to match the end of string. I'm not sure why the order matters, others knowing the dbparser codebase better probably have an answer off the top of their heads. I would assume that dbparser should always do a best match and not linear evaluation, so I'd say this is not intended and is worth a bugreport. The above is also valid for the program pattern, however, you can use parsers in that pattern as well, and the SET parser should be more performant than a regex. Balint On 10/31/2012 05:01 PM, Evan Rempel wrote:
I am having a problem with the dbparser when messages have identical leading portions.
In my case I have two messages
xlog: backup pg_xlog/000000010000014700000076 xlog: backup pg_xlog/000000010000014700000076 failed
The first of these two messages is a success, and should be safely ignored. The second of these two messages is a failure, and I want to tag it and route it to a ticket creation program so that our Operations Center can investigate.
Patterns are;
xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed
If these patterns are placed into the pattern database in this order, then the first pattern will match the failed log message, which is completely wrong. I would ignore the message rather than route it to my ticketing system.
If I change the order of them, then all works correctly.
I don't think that "hand tuning" the xml file is the correct approach, and in my case can not actually be done. My patterndb.xml file is created by a program that reads all of the patterns and tests messages from an external database.
I've worked around this by sorting the patterns descending by the length of the patterns, but if I had the two patterns;
xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@ESTRING:: @failed
then that "trick" does not work either.
As I see it, the dbparser should match complete messages only. That would make the order completely irrelevant.
Incidentally, I have the same problem with the "program" matching that the dbparser users.
Take the mimedefang program as an example. There are multiple components with program names of;
mimedefang mimedefang-multiplexor mimedefang.pl
but the program mimedefang will match them all. This means that the pattern database file must either;
1. Have all of the message patterns under mimedefang 2. have the program sections sorted in reverse order so that the shortest program matches last.
Again, I think that the pattern database should match the complete program name. Using multiple patterns for the program name is technically sufficient, but it would be nice to use a regular expression so that I can match
imapd imapds
with the program impads?
As usual, all comments welcome, especially those that explain the history of why it was done this way and how the issue can be addressed.
Thanks to everyone for such an active community.
Evan. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq