Hi Bazsi,

Continuing the conversation on the previous mail:

My guess is: because in one ruleset you only have one program whereas in the other you have multiple. Make sure the list of programs are the same.

>> This  default.xml is failing while loading :


<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
        <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
            <patterns>
                <pattern>foo</pattern>
            </patterns>
        </rule>
        <rule id="a681963842014480a83a2a2e38875439">
            <patterns>
                <pattern>anything</pattern>
            </patterns>
        </rule>
    </rules>
</ruleset>
<ruleset id="17fae6edff32a53f9f294ab21240fc2641e7a4db" description="">
       <pattern>ProxySG</pattern>
       <pattern>proxysg</pattern>
    <rules>
       <rule id="bb169f917216467985cc16e28015f5fa">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
        <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule>
    </rules>
</ruleset>

whereas, if I change the order of program pattern in the second ruleset(17fae6edff32a53f9f294ab21240fc2641e7a4db) it is loading correctly irrespective of the list of programs are different in both the rulesets.

<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
        <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
            <patterns>
                <pattern>foo</pattern>
            </patterns>
        </rule>
        <rule id="a681963842014480a83a2a2e38875439">
            <patterns>
                <pattern>anything</pattern>
            </patterns>
        </rule>
    </rules>
</ruleset>
<ruleset id="17fae6edff32a53f9f294ab21240fc2641e7a4db" description="">
      <pattern>proxysg</pattern>
       <pattern>ProxySG</pattern>
    <rules>
       <rule id="bb169f917216467985cc16e28015f5fa">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
        <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule>
    </rules>
</ruleset>

It is going very confusing now.Can you please clarify the exact behavior and how should I fix the default.xml to make it load correctly ?

Thanks,
Nitish

On Fri, Feb 21, 2020 at 3:59 PM Nitish Saboo <nitish.saboo55@gmail.com> wrote:
Hi Bazsi,

Following is the snippet from from default.xml that is failing :

<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
        <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
            <patterns>
                <pattern>foo</pattern>
            </patterns>
        </rule>
        <rule id="a681963842014480a83a2a2e38875439">
            <patterns>
                <pattern>anything</pattern>
            </patterns>
        </rule>
    </rules>
</ruleset>
<ruleset id="17fae6edff32a53f9f294ab21240fc2641e7a4db" description="">
       <pattern>ProxySG</pattern>
       <pattern>proxysg</pattern>
    <rules>
       <rule id="bb169f917216467985cc16e28015f5fa">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
        <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule>
    </rules>
</ruleset>

2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/home/nsaboo/abc/default.xml', error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'.

According to Evan and Attila these are the possible solutions:

1)<?xml version='1.0' encoding='UTF-8'?>
    <patterndb version="4">
        <ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
            <pattern>proxysg</pattern>
            <pattern>ProxySG</pattern>
            <rules>
                <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
                    <patterns>
                        <pattern>foo</pattern>
                    </patterns>
                </rule>
                <rule id="a681963842014480a83a2a2e38875439">
                    <patterns>
                        <pattern>anything</pattern>
                    </patterns>
                </rule>
                <rule id="bb169f917216467985cc16e28015f5fa">
                    <patterns>
                        <pattern>bar</pattern>
                    </patterns>
                </rule>
                <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
                    <patterns>
                        <pattern>something</pattern>
                    </patterns>
                </rule>
            </rules>
        </ruleset>
    </patterndb>



2)<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
        <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
            <patterns>
                <pattern>foo</pattern>
            </patterns>
        </rule>
        <rule id="a681963842014480a83a2a2e38875439">
            <patterns>
                <pattern>anything</pattern>
            </patterns>
        </rule>
       <rule id="bb169f917216467985cc16e28015f598">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
       <rule id="94d4a0c324c8-44a88cf3d4640477d399">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule>
    </rules>
</ruleset>
<ruleset id="17fae6edff32a53f9f294ab21240fc2641e7a4db" description="">
    <pattern>ProxySG</pattern>
    <rules>
       <rule id="bb169f917216467985cc16e28015f5fa">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
        <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule>
    </rules>
 </ruleset>


Then what is the issue in my default.xml ?

My guess is: because in one ruleset you only have one program whereas in the other you have multiple. Make sure the list of programs are the same.

>>Are you saying if a program pattern is part of multiple rulesets , then both the rulesets should have same sets of program patterns or same number of program pattern ?

Thanks,
Nitish

On Fri, Feb 21, 2020 at 3:39 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
My guess is: because in one ruleset you only have one program whereas in the other you have multiple. Make sure the list of programs are the same.

I dont have source code nor the abioity to try it at the moment, as I am standing on the top of a hill in the Alps :)

On Fri, Feb 21, 2020, 10:59 Nitish Saboo <nitish.saboo55@gmail.com> wrote:
Hi Evan, Attila,

I got the following Note from the following link.I have made the line bold.This is from syslog-ng Adminitrator guide from 3.25 version

https://support.oneidentity.com/technical-documents/syslog-ng-open-source-edition/3.25/administration-guide/77#TOPIC-1349587

NOTE:
If the ${PROGRAM} part of a message is empty, rules with an empty Program Pattern are used to classify the message.

If the same Program Pattern is used in multiple rulesets, the rules of these rulesets are merged, and every rule is used to classify the message. Note that message patterns must be unique within the merged rulesets, but the currently only one ruleset is checked for uniqueness.

If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.


1) According to that line( bold one) you should accept multiple program pattern in different rulesets ..Am i correct? Please correct me if I am missing something here.

2) If my understanding is correct for [1], then why is my default.xml failing with the following error while loading:

2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/home/nsaboo/abc/default.xml', error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'.

Thanks,
Nitish

On Tue, Feb 18, 2020 at 10:47 AM Nitish Saboo <nitish.saboo55@gmail.com> wrote:
Thank you Evan for you response.Looks like changes in .xml file is the only option.

Thanks,
Nitish

On Mon, Feb 17, 2020 at 10:20 PM Evan Rempel <erempel@uvic.ca> wrote:
Syslog messages have multiple components to the. Date/time, host, program, message and a few others.
The important point is that the program and message are independent items. This means that the pattern database must use a two phase matching method, first matching the program and then matching the message.

Another way to look at it is that the same message for two different programs means two independent matching rules.

Taking this into account you need to write your patterndb differently. Either merging the rulesets (as proposed by aszakacs), or breaking apart the duplicate rule (one rule for two programs) as shown below. Although it may have worked (or appeared to have worked) in syslog-ng 3.6 or 3.7, it was against the patterndb specification.

<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
        <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
            <patterns>
                <pattern>foo</pattern>
            </patterns>
        </rule>
        <rule id="a681963842014480a83a2a2e38875439">
            <patterns>
                <pattern>anything</pattern>
            </patterns>
        </rule>
       <rule id="bb169f917216467985cc16e28015f598">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
       <rule id="94d4a0c324c8-44a88cf3d4640477d399">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule> 
    </rules>
</ruleset>
<ruleset id="17fae6edff32a53f9f294ab21240fc2641e7a4db" description="">
    <pattern>ProxySG</pattern>
    <rules>
       <rule id="bb169f917216467985cc16e28015f5fa">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
        <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule>
    </rules>
 </ruleset>



Both this solution and the one by aszakacs have ONE ruleset for any given program (ruleset pattern).

I don't see any other alternatives.

Evan.


On 2/17/20 1:29 AM, Nitish Saboo wrote:

On Mon, Feb 17, 2020 at 2:37 PM Nitish Saboo <nitish.saboo55@gmail.com> wrote:
Hi Attila,

1,2) The issue is that you used "proxysg" in two different rulesets. I believe syslog-ng could handle this, but we do not handle it right now.
>> But the same default.xml file is working fine in version syslog-ng-3,6,2 and syslog-ng-3.7.1 where we have same program in different rulesets.The behavior in version syslog-ng-3,6,2 and syslog-ng-3.7.1 is the expected behavior or is the current behavior in syslog-ng-3.25.1 version is having a Bug ?

3/4) You can merge the two rulesets, if it is okay for you:
>>Merging the rulesets is not possible at this moment.
Do we have any workaround for this scenario other than merging the rulesets?
When can we expect a fix for this in near future ?

Thanks,
Nitish

On Mon, Feb 17, 2020 at 2:23 PM Attila Szakacs (aszakacs) <Attila.Szakacs@oneidentity.com> wrote:
Hi!

1,2) The issue is that you used "proxysg" in two different rulesets. I believe syslog-ng could handle this, but we do not handle it right now.
3/4) You can merge the two rulesets, if it is okay for you:

<?xml version='1.0' encoding='UTF-8'?>
    <patterndb version="4">
        <ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
            <pattern>proxysg</pattern>
            <pattern>ProxySG</pattern>
            <rules>
                <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
                    <patterns>
                        <pattern>foo</pattern>
                    </patterns>
                </rule>
                <rule id="a681963842014480a83a2a2e38875439">
                    <patterns>
                        <pattern>anything</pattern>
                    </patterns>
                </rule>
                <rule id="bb169f917216467985cc16e28015f5fa">
                    <patterns>
                        <pattern>bar</pattern>
                    </patterns>
                </rule>
                <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
                    <patterns>
                        <pattern>something</pattern>
                    </patterns>
                </rule>
            </rules>
        </ruleset>
    </patterndb>

Regards,
Attila

From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Nitish Saboo <nitish.saboo55@gmail.com>
Sent: Monday, February 17, 2020 6:30 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Subject: Re: [syslog-ng] Warnings and error while loading default.xml in syslog-ng-3.25.1
 
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Evan,

Apologies for the confusion but I did close the rule tag in my default.xml , missed adding it here.
The error does not seem to be related to improper closing of tags.

This is how it looks :

<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
        <rule id="f1e2bfd7bb85402a88d0b732821a0f94">
            <patterns>
                <pattern>foo</pattern>
            </patterns>
        </rule>
        <rule id="a681963842014480a83a2a2e38875439">
            <patterns>
                <pattern>anything</pattern>
            </patterns>
        </rule>
    </rules>
</ruleset>
<ruleset id="17fae6edff32a53f9f294ab21240fc2641e7a4db" description="">
       <pattern>ProxySG</pattern>
       <pattern>proxysg</pattern>
    <rules>
       <rule id="bb169f917216467985cc16e28015f5fa">
          <patterns>
             <pattern>bar</pattern>
          </patterns>
       </rule>
        <rule id="94d4a0c324c8-44a88cf3d4640477d35e">
          <patterns>
             <pattern>something</pattern>
          </patterns>
       </rule>
    </rules>
 </ruleset>


I am getting the following error message:

 2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/home/nsaboo/abc/default.xml', error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'.

My hunch is the issue is related to merging of two rulesets but I am not able to understand why is there mismatching of program names.

 1) Can someone please help me understand the issue here ?

 2) Is the issue seen because a ruleset has multiple programs in it or is it because the same program 'proxysg' is being used in different rulesets ?

 3) From the above snippet of default.xml, what changes can I make into default.xml to avoid the error ?

 4) Is there a workaround for this issue ?

 Thanks,
 Nitish


On Sun, Feb 16, 2020 at 12:40 AM Evan Rempel <erempel@uvic.ca> wrote:
I'm not exactly sure what is or is not permitted in the pattern database but I two comments.

1. you need end your rule tag and your rules tag before you start a new ruleset tag.

2. What I do in my pattern database is of the form.

<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
        <rule id="f582419b3baa42d4a57e42b89704e38c">
            <patterns>
                <pattern>foo</pattern>
            </patterns>
        </rule>
        <rule id="bb169f917216467985cc16e28015f5fa">
            <patterns>
                <pattern>bar</pattern>
            </patterns>
    </rules>
</ruleset>


Note:
1. the closing tag of </rule> before a new starting tag of <rule>
2. Multiple "rule" entries inside the the "rules" entry.
3. the closing tag of </rule> before the closing tag of </rules>
4. the closing tag of </rules> before the closing tag of </ruleset>

I hope that helps.

Evan.

On 2/15/20 12:43 AM, Nitish Saboo wrote:
Hi,

After debugging further into the issue looks like there was a fix for patterndb rule clash in syslog-ng-3.8 and this is the commit-id '12cd960c8f47260b0b0d4154b096994d66fe345'
for the fix. And for this reason I am getting the following error for same default.xml in syslog-ng-3.25.1 version and not in syslog-ng3.6.2 and syslog-ng3.7.1.

2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/home/nsaboo/abc/default.xml', error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'.

Snippet from default.xml
==========================

<ruleset id="f582419b3baa42d4a57e42b89704e38c" description="">
    <pattern>proxysg</pattern>
    <rules>
      <rule id="f582419b3baa42d4a57e42b89704e38c">
      <patterns>
         <pattern>foo</pattern>
      </patterns>
 
  <ruleset id="8d633c824e844a559088d803464e507a" description="">
    <pattern>ProxySG</pattern>
    <pattern>proxysg</pattern>
    <rules>
      <rule id="bb169f917216467985cc16e28015f5fa">
      <patterns>
         <pattern>bar</pattern>
      </patterns>

 I am not able to understand the error message clearly.

 1) Can someone please help me understand the issue here ?

 2) Is the issue seen because a ruleset has multiple programs in it or is it because the same program 'proxysg' is being used in different rulesets ?

 3) From the above snippet of default.xml, what changes can I make into default.xml to avoid the error ?

 4) Is there a workaround for this issue ?

 Thanks,
 Nitish

On Fri, Feb 14, 2020 at 2:40 PM Nitish Saboo <nitish.saboo55@gmail.com> wrote:
Hi Attila,

Thanks for your response.

And what about the following error:

2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/home/nsaboo/abc/default.xml', error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'.

The same default.xml file was getting loaded correctly in syslog-ng-3.6.2 and syslog-ng-3.7.1 but getting following error while loading same default.xml in syslog-ng3.25.1 

I came across a similar issue on githib 'https://github.com/syslog-ng/syslog-ng/issues/2763' .I see the issue is still in open state.Is there a workaround for this issue?

Thanks,
Nitish

On Fri, Feb 14, 2020 at 1:12 PM Attila Szakacs (aszakacs) <Attila.Szakacs@oneidentity.com> wrote:
Hi!

WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file;

^^^ This refers to the syslog-ng.conf file version.

The correct way to resolve it, and fix the buggy behavior of != and ==, should be to change the != operators between strings to neq in your filters.

Regards,
Attila
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Nitish Saboo <nitish.saboo55@gmail.com>
Sent: Thursday, February 13, 2020 12:17 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Subject: [syslog-ng] Warnings and error while loading default.xml in syslog-ng-3.25.1
 
Hi,

I am using syslog-ng version 3.25.1.Getting following Warnings and error while initialising syslog-ng engine:

[2020-02-13T10:47:29.627899] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file;
[2020-02-13T10:47:29.627968] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file;
[2020-02-13T10:47:29.628059] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file;
[2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/opt/tap-parsing/patterns/default.xml', error='/opt/tap-parsing/patterns/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'


1)For the following warnings, to which version I have to bump up the configuration file ?

2020-02-13T10:47:29.627899] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file;
[2020-02-13T10:47:29.627968] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file;
[2020-02-13T10:47:29.628059] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file;

Currrently the configuration version is the following:

configuration = cfg_new(0x0302)

Do I have to change it to '0x0319' as defined in 'lib/versioning.h' ?

2)The same default.xml file was getting loaded correctly in syslog-ng-3.6.2 and syslog-ng-3.7.1 but getting following error while loading same default.xml in syslog-ng3.25.1 


2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/home/nsaboo/abc/default.xml', error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'.

What can be the reason for this error ?


Thanks,

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq