[syslog-ng] Need to parse named query log entries and get just the domain portion
Jim Hendrick
james.r.hendrick at gmail.com
Tue Oct 31 17:55:23 UTC 2017
The shell is parsing your command line - look at it from that perspective
and I think you will see why the quotes fixes it.
On Tue, Oct 31, 2017 at 5:42 PM, Robert Webb <rwebb at ropeguru.com> wrote:
> Circling back around to this question and another log entry having issues.
>
>
>
> So I have some custom logging happening on a server using tcpdump.
> Basically, tcpdump is running and when a hit takes place, the data is sent
> to syslog via the logger command. This part is working and the entries are
> making it to syslog-ng.
>
>
>
> I have used pdbtool to test my xml file and all works. I even verified
> that the tag is correct so the pattern will match.
>
>
>
> What I have a problem with is in testing with pdbtool, if I use the
> standard double quote, I get the following:
>
>
>
> pdbtool match -D -c -p /etc/syslog-ng/patterndb.d/sinkhole.xml -P
> "sinkhole" -M " www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp
> cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co.
> A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS
> 140.xxx.xxx.xxx.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1,
> 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A
> 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)"
>
> -bash: !]: event not found
>
>
>
> However, if I use single quotes, I get what I expect:
>
>
>
> pdbtool match -D -c -p /etc/syslog-ng/patterndb.d/sinkhole.xml -P
> "sinkhole" -M ' www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp
> cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co.
> A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS
> 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1,
> 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A
> 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)'
>
> Pattern matching part:
>
> @ESTRING:None= www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp
> cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co.
> A 127.0.0.1 @ @HOSTNAME:sinkhole.domain.name=cadhya.co.@@ANYSTRING:None=
> NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.172.111.111.cadhya.co.
> ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co.
> AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1,
> 140.172.111.111.cadhya.co. AAAA ::1 (192)@
>
> Matching part:
>
> www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79
> -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1
> ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS
> 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1,
> 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A
> 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)
>
> Values:
>
> MESSAGE= www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum
> 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A
> 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS
> 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1,
> 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A
> 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)
>
> PROGRAM=sinkhole
>
> .classifier.class=system
>
> .classifier.rule_id=21071031002
>
> sinkhole.domain.name=cadhya.co.
>
> TAGS=.classifier.system,sinkhole,queries
>
>
>
> *From:* Robert Webb
> *Sent:* Friday, October 27, 2017 10:56 AM
> *To:* Evan Rempel <erempel at uvic.ca>
> *Cc:* syslog-ng at lists.balabit.hu
> *Subject:* RE: [syslog-ng] Need to parse named query log entries and get
> just the domain portion
>
>
>
> Evan,
>
>
>
> Thanks you for pointing me in the right direction. I was so focused on the
> matching starting @ “queries: info:” that I totally forgot that the first
> part of the actual MSG has the date and time sent with it.
>
>
>
> My pattern now takes that into account.
>
>
>
> Message: 27-Oct-2017 14:19:42.359 queries: info: client
> 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E
> (137.75.75.75)
>
>
>
> <pattern>@ESTRING:: @@ESTRING:: @queries: info: client
> @IPvANY:named.client.ip@#@NUMBER:named.client.port:@
> @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@
> @STRING:named.record.type:@ @ESTRING:named.record.flags:
> @(@IPvANY:named.server.ip@)</pattern>
>
>
>
> Robert
>
>
>
> *From:* Evan Rempel [mailto:erempel at uvic.ca <erempel at uvic.ca>]
> *Sent:* Friday, October 27, 2017 9:52 AM
> *To:* Robert Webb <rwebb at ropeguru.com>
> *Subject:* Re: [syslog-ng] Need to parse named query log entries and get
> just the domain portion
>
>
>
> In the past when I have had this problem it was in the patterndb portion
> just outside if what you have included.
>
> essentially the program does not match, so the rule is never attempted.
> This results in none of the tag-value pairs being populated. The pdbtool
> shows a match because the pattern is always in the correct program.
>
> Your patterndb portion is inside of a set of a config such as
>
> <patterndb version="4" pub_date="2009-09-01">
> <ruleset name="named" id="some-unique-value">
> <pattern>named</pattern>
> <rules>
>
> [ your included pattern db data ]
>
> </ruleset>
> </pattendb>
>
> It is the line
> <pattern>named</pattern>
> that matches the parsed program name from the syslog line to allow the
> rule to be parsed against the
> syslog message.
>
> For testing purposes I make a destination to write a json object. That way
> I can see everything that populated.
>
> template t_json { template("$(format-json --scope everything )\n");
> template_escape(no); };
> destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000"
> template(t_json)); };
>
> Evan.
>
> On 10/27/2017 05:22 AM, Robert Webb wrote:
>
> So I have a situation where I have a syslog feed coming in that is sending
> named (BIND9) query logs. On the DNS servers themselves, we are doing sink
> hole via a local zone file and sending the dns response back as 127.0.0.1.
>
>
>
> Since there is no way for BIND9 to give me logging of the DNS query
> response back to the user, I need to be able to take a query log entry,
> parse it to get just the domain portion, and then match the domain to a
> list. I am trying to do all this inside syslog-ng.
>
>
>
> I have been given some guidance to use a pattern database. I worked with
> this and am successfully matching my incoming query log data. Using the
> pdbtool, I get a full match and a list of the values I expect. But where I
> run into an issue after searching through all the documentation is, where
> can these values be used? I tried building a template against a
> destination, and none of the values populate. So I am stuck at this point.
>
>
>
> I have also been told that I may have to do a custom python parser instead.
>
>
>
> Any help or guidance is appreciated.
>
>
>
> Here is my rule:
>
>
>
> <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee"
> class="system">
>
> <patterns>
>
> <pattern>queries: info: client @IPvANY:named.client.ip@#@
> NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query:
> @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@
> @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern>
>
> </patterns>
>
> <examples>
>
> <example>
>
> <test_message program="named">client 213.46.246.51#1890:
> query: tumtali.fegade.com IN A - (103.5.198.211)</test_message>
>
> <test_values>
>
> <test_value name="named.client.ip">213.46.
> 246.51</test_value>
>
> <test_value name="named.client.port">1890</test_value>
>
> <test_value name="named.domain.name">tumtali.fegade.com
> </test_value>
>
> <test_value name="named.record.type">A</test_value>
>
> <test_value name="named.record.flags">-</test_value>
>
> <test_value name="named.server.ip">103.5.
> 198.211</test_value>
>
> </test_values>
>
> </example>
>
> <example>
>
> <test_message program="named">client 10.187.195.186#40441:
> query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message>
>
> <test_values>
>
> <test_value name="named.client.ip">10.187.
> 195.186</test_value>
>
> <test_value name="named.client.port">40441</test_value>
>
> <test_value name="named.domain.name">
> banister.paars.tld</test_value>
>
> <test_value name="named.record.type">A</test_value>
>
> <test_value name="named.record.flags">-EDC</test_value>
>
> <test_value name="named.server.ip">10.226.
> 134.25</test_value>
>
> </test_values>
>
> </example>
>
> <example>
>
> <test_message program="named">queries: client
> 192.168.102.189#58454: query: maps.googleapis.com IN A +
> (192.168.102.2)</test_message>
>
> <test_values>
>
> <test_value name="named.client.ip">192.
> 168.102.189</test_value>
>
> <test_value name="named.client.port">58454</test_value>
>
> <test_value name="named.domain.name">maps.googleapis.com
> </test_value>
>
> <test_value name="named.record.type">A</test_value>
>
> <test_value name="named.record.flags">+</test_value>
>
> <test_value name="named.server.ip">192.
> 168.102.2</test_value>
>
> </test_values>
>
> </example>
>
> </examples>
>
> <values>
>
> </values>
>
> <tags>
>
> <tag>named</tag>
>
> <tag>queries</tag>
>
> </tags>
>
> </rule>
>
>
>
>
>
>
>
>
>
> Here are test results:
>
>
>
> Pattern matching part:
>
> queries: info: client @IP:named.client.ip=137.75.76.138@
> #@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@
> query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@
> @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@
> (@IP:named.server.ip=137.75.75.75@)
>
> Matching part:
>
> queries: info: client 137.75.76.138#38643 (sr.symcb.com): query:
> sr.symcb.com IN A +E (137.75.75.75)
>
> Values:
>
> MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com): query:
> sr.symcb.com IN A +E (137.75.75.75)
>
> PROGRAM=named
>
> .classifier.class=system
>
> .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee
>
> named.client.ip=137.75.76.138
>
> named.client.port=38643
>
> named.domain.name=(sr.symcb.com)
>
> named.record.type=A
>
> named.server.ip=137.75.75.75
>
> named.host.name=sr.symcb.com
>
> named.record.flags=+E
>
> TAGS=.classifier.system,named,queries
>
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20171031/aea8230e/attachment-0001.html>
More information about the syslog-ng
mailing list