[syslog-ng] Diagnosing patterndb load errors

[Balazs Scheidler]

Hi,

On Tue, 2010-10-12 at 18:41 -0700, Matthew Hall wrote:
> I am getting the following sequence of load errors for my patterndbs.
> 
> Duplicates eliminated, timestamps zeroed out, host info anonymized.
> 
> Has anybody else gotten these and learned how to fix them?
> 
> I'm having a hard time figuring out what 'type', 'name', and 'value' mean,
> to track them down in the XML files.
> 
> Thanks,
> Matthew Hall.
> 
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Name-value pairs cannot have a zero-length name; value=''
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='Neighbor Down: Dead timer expired', value='19'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='26'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='28'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='34'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='36'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='39'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='40'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='41'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='44'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=', domain = <not specified>', value='8'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='12'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='13'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='20'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='21'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='22'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='23'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='24'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='25'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='26'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='27'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='28'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='29'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='30'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='31'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='32'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='33'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='34'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='35'
> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='4', name='None', value='38'

As it seems I'd do have to work somewhat on these error messages. Even
though I didn't write them in the first place, Marci is to be blamed :)

But anyway, trying to be helpful here:

As you probably know the patterns are loaded into a so called radix tree
in order to scale to the domain of 10000s of patterns without
sacrificing performance too much.

http://en.wikipedia.org/wiki/Radix_tree

The error message states that there's a conflict in the rules being
loaded. A conflict can happen when there's nothing that differentiates
the two rules for syslog-ng.

So let's consider this example:

<rule id="1"><pattern>foobar @STRING:reason@</pattern></rule>
<rule id="2"><pattern>foobar @NUMBER:int@</pattern></rule>

These two pattern would result in the following RADIX tree:


          / @STRING:reason@
         /
|foobar |
         \
          \ @NUMBER:int@

When evaluating the incoming message, syslog-ng will try to match the
rules in the order it has encountered them and the first parser which
matches will "win".

However if you have two rules:

<rule id="1"><pattern>foobar @STRING:reason@</pattern></rule>
<rule id="2"><pattern>foobar @STRING:reason@</pattern></rule>

If you try to load these two patterns, a conflict occurs, syslog-ng is
unable to differentiate between the two rules, they are the same. If the
rules would continue this way:

<rule id="1"><pattern>foobar @STRING:reason@ foo as hell</pattern></rule>
<rule id="2"><pattern>foobar @STRING:reason@ bar as hell</pattern></rule>

This would make syslog-ng happy, because after the parser there's key 
information left that differentiates the two rules. Of course if 
that's again equal you'd get a similar message about duplication.

(NOTE: patterndb conflict handling could be improved, but so far
the literal text in the patterns were enough to avoid conflicts 
in the first place)

Back to what the message means:

Duplicate parser node in radix tree; type='5', name='reason', value='25'

  * type: is the identifier of the parser, defined in radix.h in the enum 
    listing RPT_ entries. type==5 means NUMBER parser
  * name: the name of the name-value entry to be parsed out
  * value: is the ID of the rule that failed to be inserted

Duplicate key in radix tree; key=' packet', value='34'

  * this time it wasn't the parser but the literal text that conflicted 
    at rule id=34

Hope this helps.

-- 
Bazsi