[syslog-ng] dbparser problem isolating messages

[Evan Rempel]

I am having a problem with the dbparser when messages have identical leading portions.

In my case I have two messages

xlog: backup pg_xlog/000000010000014700000076
xlog: backup pg_xlog/000000010000014700000076 failed


The first of these two messages is a success, and should be safely ignored.
The second of these two messages is a failure, and I want to tag it and route it to a ticket creation program
so that our Operations Center can investigate.

Patterns are;

xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@
xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed

If these patterns are placed into the pattern database in this order, then the first pattern will match the failed log
message, which is completely wrong. I would ignore the message rather than route it to my ticketing system.

If I change the order of them, then all works correctly.

I don't think that "hand tuning" the xml file is the correct approach, and in my case can not actually be done.
My patterndb.xml file is created by a program that reads all of the patterns and tests messages from an external database.

I've worked around this by sorting the patterns descending by the length of the patterns, but if I had the two patterns;

xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@
xlog: backup pg_xlog/@ESTRING:: @failed

then that "trick" does not work either.

As I see it, the dbparser should match complete messages only. That would make the order completely irrelevant.







Incidentally, I have the same problem with the "program" matching that the dbparser users.

Take the mimedefang program as an example. There are multiple components with program names of;

mimedefang
mimedefang-multiplexor
mimedefang.pl

but the program mimedefang will match them all. This means that the pattern database file must either;

1. Have all of the message patterns under mimedefang
2. have the program sections sorted in reverse order so that the shortest program matches last.

Again, I think that the pattern database should match the complete program name. Using multiple patterns for the program 
name is technically sufficient, but it would be nice to use a regular expression so that I can match

imapd
imapds

with the program
impads?


As usual, all comments welcome, especially those that explain the history of why it was done this way and
how the issue can be addressed.


Thanks to everyone for such an active community.

Evan.