[syslog-ng] Diagnosing patterndb load errors

[Martin Holste]

Ah, that's perfectly clear now.  I thought it was something along the
lines of pattern collision.  In the IDS world, this is a common
problem and is handled in Snort by executing on all matches up to a
configured limit (8 by default).  I think the larger question is can a
log be classified two ways at the same time, and if so, should it be
copied as it works its way through the log chain for each full pattern
it matches?  I think the warning is probably good enough for most
people.

On Fri, Oct 15, 2010 at 3:47 PM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> Hi,
>
> On Tue, 2010-10-12 at 18:41 -0700, Matthew Hall wrote:
>> I am getting the following sequence of load errors for my patterndbs.
>>
>> Duplicates eliminated, timestamps zeroed out, host info anonymized.
>>
>> Has anybody else gotten these and learned how to fix them?
>>
>> I'm having a hard time figuring out what 'type', 'name', and 'value' mean,
>> to track them down in the XML files.
>>
>> Thanks,
>> Matthew Hall.
>>
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Name-value pairs cannot have a zero-length name; value=''
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='Neighbor Down: Dead timer expired', value='19'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='26'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='28'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='34'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='36'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='39'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='40'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='41'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='44'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate key in radix tree; key=', domain = <not specified>', value='8'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='12'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='13'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='20'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='21'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='22'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='23'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='24'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='25'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='26'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='27'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='28'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='29'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='30'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='31'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='32'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='33'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='34'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='35'
>> Jan 01 00:00:00 s_internal at host syslog-ng[23005]: Duplicate parser node in radix tree; type='4', name='None', value='38'
>
> As it seems I'd do have to work somewhat on these error messages. Even
> though I didn't write them in the first place, Marci is to be blamed :)
>
> But anyway, trying to be helpful here:
>
> As you probably know the patterns are loaded into a so called radix tree
> in order to scale to the domain of 10000s of patterns without
> sacrificing performance too much.
>
> http://en.wikipedia.org/wiki/Radix_tree
>
> The error message states that there's a conflict in the rules being
> loaded. A conflict can happen when there's nothing that differentiates
> the two rules for syslog-ng.
>
> So let's consider this example:
>
> <rule id="1"><pattern>foobar @STRING:reason@</pattern></rule>
> <rule id="2"><pattern>foobar @NUMBER:int@</pattern></rule>
>
> These two pattern would result in the following RADIX tree:
>
>
>          / @STRING:reason@
>         /
> |foobar |
>         \
>          \ @NUMBER:int@
>
> When evaluating the incoming message, syslog-ng will try to match the
> rules in the order it has encountered them and the first parser which
> matches will "win".
>
> However if you have two rules:
>
> <rule id="1"><pattern>foobar @STRING:reason@</pattern></rule>
> <rule id="2"><pattern>foobar @STRING:reason@</pattern></rule>
>
> If you try to load these two patterns, a conflict occurs, syslog-ng is
> unable to differentiate between the two rules, they are the same. If the
> rules would continue this way:
>
> <rule id="1"><pattern>foobar @STRING:reason@ foo as hell</pattern></rule>
> <rule id="2"><pattern>foobar @STRING:reason@ bar as hell</pattern></rule>
>
> This would make syslog-ng happy, because after the parser there's key
> information left that differentiates the two rules. Of course if
> that's again equal you'd get a similar message about duplication.
>
> (NOTE: patterndb conflict handling could be improved, but so far
> the literal text in the patterns were enough to avoid conflicts
> in the first place)
>
> Back to what the message means:
>
> Duplicate parser node in radix tree; type='5', name='reason', value='25'
>
>  * type: is the identifier of the parser, defined in radix.h in the enum
>    listing RPT_ entries. type==5 means NUMBER parser
>  * name: the name of the name-value entry to be parsed out
>  * value: is the ID of the rule that failed to be inserted
>
> Duplicate key in radix tree; key=' packet', value='34'
>
>  * this time it wasn't the parser but the literal text that conflicted
>    at rule id=34
>
> Hope this helps.
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>