[syslog-ng] Need to parse named query log entries and get just the domain portion

Fri Oct 27 14:56:28 UTC 2017

Evan,

Thanks you for pointing me in the right direction. I was so focused on the matching starting @ "queries: info:" that I totally forgot that the first part of the actual MSG has the date and time sent with it.

My pattern now takes that into account.

Message: 27-Oct-2017 14:19:42.359 queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)

<pattern>@ESTRING:: @@ESTRING:: @queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern>

Robert

From: Evan Rempel [mailto:erempel at uvic.ca]
Sent: Friday, October 27, 2017 9:52 AM
To: Robert Webb <rwebb at ropeguru.com>
Subject: Re: [syslog-ng] Need to parse named query log entries and get just the domain portion

In the past when I have had this problem it was in the patterndb portion just outside if what you have included.

essentially the program does not match, so the rule is never attempted. This results in none of the tag-value pairs being populated. The pdbtool shows a match because the pattern is always in the correct program.

Your patterndb portion is inside of a set of a config such as

<patterndb version="4" pub_date="2009-09-01">
   <ruleset name="named" id="some-unique-value">
      <pattern>named</pattern>
      <rules>

[ your included pattern db data ]

   </ruleset>
</pattendb>

It is the line
      <pattern>named</pattern>
that matches the parsed program name from the syslog line to allow the rule to be parsed against the
syslog message.

For testing purposes I make a destination to write a json object. That way I can see everything that populated.

template t_json { template("$(format-json --scope everything )\n"); template_escape(no); };
destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000" template(t_json)); };

Evan.

On 10/27/2017 05:22 AM, Robert Webb wrote:
So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1.

Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng.

I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point.

I have also been told that I may have to do a custom python parser instead.

Any help or guidance is appreciated.

Here is my rule:

      <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system">
        <patterns>
          <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern>
        </patterns>
        <examples>
          <example>
            <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message>
            <test_values>
              <test_value name="named.client.ip">213.46.246.51</test_value>
              <test_value name="named.client.port">1890</test_value>
              <test_value name="named.domain.name">tumtali.fegade.com</test_value>
              <test_value name="named.record.type">A</test_value>
              <test_value name="named.record.flags">-</test_value>
              <test_value name="named.server.ip">103.5.198.211</test_value>
            </test_values>
          </example>
          <example>
            <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message>
            <test_values>
              <test_value name="named.client.ip">10.187.195.186</test_value>
              <test_value name="named.client.port">40441</test_value>
              <test_value name="named.domain.name">banister.paars.tld</test_value>
              <test_value name="named.record.type">A</test_value>
              <test_value name="named.record.flags">-EDC</test_value>
              <test_value name="named.server.ip">10.226.134.25</test_value>
            </test_values>
          </example>
          <example>
            <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message>
            <test_values>
              <test_value name="named.client.ip">192.168.102.189</test_value>
              <test_value name="named.client.port">58454</test_value>
              <test_value name="named.domain.name">maps.googleapis.com</test_value>
              <test_value name="named.record.type">A</test_value>
              <test_value name="named.record.flags">+</test_value>
              <test_value name="named.server.ip">192.168.102.2</test_value>
            </test_values>
          </example>
        </examples>
        <values>
        </values>
        <tags>
          <tag>named</tag>
          <tag>queries</tag>
        </tags>
      </rule>

Here are test results:

Pattern matching part:
queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@)
Matching part:
queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)
Values:
MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)
PROGRAM=named
.classifier.class=system
.classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee
named.client.ip=137.75.76.138
named.client.port=38643
named.domain.name=(sr.symcb.com)
named.record.type=A
named.server.ip=137.75.75.75
named.host.name=sr.symcb.com
named.record.flags=+E
TAGS=.classifier.system,named,queries

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20171027/89c0661b/attachment.html>