Ah, that's perfectly clear now. I thought it was something along the lines of pattern collision. In the IDS world, this is a common problem and is handled in Snort by executing on all matches up to a configured limit (8 by default). I think the larger question is can a log be classified two ways at the same time, and if so, should it be copied as it works its way through the log chain for each full pattern it matches? I think the warning is probably good enough for most people. On Fri, Oct 15, 2010 at 3:47 PM, Balazs Scheidler <bazsi@balabit.hu> wrote:

Hi,

On Tue, 2010-10-12 at 18:41 -0700, Matthew Hall wrote:

...

I am getting the following sequence of load errors for my patterndbs.

Duplicates eliminated, timestamps zeroed out, host info anonymized.

Has anybody else gotten these and learned how to fix them?

I'm having a hard time figuring out what 'type', 'name', and 'value' mean, to track them down in the XML files.

Thanks, Matthew Hall.

Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Name-value pairs cannot have a zero-length name; value='' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='Neighbor Down: Dead timer expired', value='19' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='36' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='39' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='40' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='41' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='44' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=', domain = <not specified>', value='8' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='12' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='13' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='20' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='21' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='22' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='23' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='24' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='27' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='29' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='30' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='31' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='32' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='33' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='35' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='4', name='None', value='38'

As it seems I'd do have to work somewhat on these error messages. Even though I didn't write them in the first place, Marci is to be blamed :)

But anyway, trying to be helpful here:

As you probably know the patterns are loaded into a so called radix tree in order to scale to the domain of 10000s of patterns without sacrificing performance too much.

http://en.wikipedia.org/wiki/Radix_tree

The error message states that there's a conflict in the rules being loaded. A conflict can happen when there's nothing that differentiates the two rules for syslog-ng.

So let's consider this example:

<rule id="1"><pattern>foobar @STRING:reason@</pattern></rule> <rule id="2"><pattern>foobar @NUMBER:int@</pattern></rule>

These two pattern would result in the following RADIX tree:

         / @STRING:reason@         / |foobar |         \          \ @NUMBER:int@

When evaluating the incoming message, syslog-ng will try to match the rules in the order it has encountered them and the first parser which matches will "win".

However if you have two rules:

<rule id="1"><pattern>foobar @STRING:reason@</pattern></rule> <rule id="2"><pattern>foobar @STRING:reason@</pattern></rule>

If you try to load these two patterns, a conflict occurs, syslog-ng is unable to differentiate between the two rules, they are the same. If the rules would continue this way:

<rule id="1"><pattern>foobar @STRING:reason@ foo as hell</pattern></rule> <rule id="2"><pattern>foobar @STRING:reason@ bar as hell</pattern></rule>

This would make syslog-ng happy, because after the parser there's key information left that differentiates the two rules. Of course if that's again equal you'd get a similar message about duplication.

(NOTE: patterndb conflict handling could be improved, but so far the literal text in the patterns were enough to avoid conflicts in the first place)

Back to what the message means:

Duplicate parser node in radix tree; type='5', name='reason', value='25'

 * type: is the identifier of the parser, defined in radix.h in the enum    listing RPT_ entries. type==5 means NUMBER parser  * name: the name of the name-value entry to be parsed out  * value: is the ID of the rule that failed to be inserted

Duplicate key in radix tree; key=' packet', value='34'

 * this time it wasn't the parser but the literal text that conflicted    at rule id=34

Hope this helps.

-- Bazsi

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html