The shell is parsing your command line - look at it from that perspective and I think you will see why the quotes fixes it. On Tue, Oct 31, 2017 at 5:42 PM, Robert Webb <rwebb@ropeguru.com> wrote:
Circling back around to this question and another log entry having issues.
So I have some custom logging happening on a server using tcpdump. Basically, tcpdump is running and when a hit takes place, the data is sent to syslog via the logger command. This part is working and the entries are making it to syslog-ng.
I have used pdbtool to test my xml file and all works. I even verified that the tag is correct so the pattern will match.
What I have a problem with is in testing with pdbtool, if I use the standard double quote, I get the following:
pdbtool match -D -c -p /etc/syslog-ng/patterndb.d/sinkhole.xml -P "sinkhole" -M " www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.xxx.xxx.xxx.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)"
-bash: !]: event not found
However, if I use single quotes, I get what I expect:
pdbtool match -D -c -p /etc/syslog-ng/patterndb.d/sinkhole.xml -P "sinkhole" -M ' www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)'
Pattern matching part:
@ESTRING:None= www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 @ @HOSTNAME:sinkhole.domain.name=cadhya.co.@@ANYSTRING:None= NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)@
Matching part:
www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)
Values:
MESSAGE= www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)
PROGRAM=sinkhole
.classifier.class=system
.classifier.rule_id=21071031002
sinkhole.domain.name=cadhya.co.
TAGS=.classifier.system,sinkhole,queries
*From:* Robert Webb *Sent:* Friday, October 27, 2017 10:56 AM *To:* Evan Rempel <erempel@uvic.ca> *Cc:* syslog-ng@lists.balabit.hu *Subject:* RE: [syslog-ng] Need to parse named query log entries and get just the domain portion
Evan,
Thanks you for pointing me in the right direction. I was so focused on the matching starting @ “queries: info:” that I totally forgot that the first part of the actual MSG has the date and time sent with it.
My pattern now takes that into account.
Message: 27-Oct-2017 14:19:42.359 queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)
<pattern>@ESTRING:: @@ESTRING:: @queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern>
Robert
*From:* Evan Rempel [mailto:erempel@uvic.ca <erempel@uvic.ca>] *Sent:* Friday, October 27, 2017 9:52 AM *To:* Robert Webb <rwebb@ropeguru.com> *Subject:* Re: [syslog-ng] Need to parse named query log entries and get just the domain portion
In the past when I have had this problem it was in the patterndb portion just outside if what you have included.
essentially the program does not match, so the rule is never attempted. This results in none of the tag-value pairs being populated. The pdbtool shows a match because the pattern is always in the correct program.
Your patterndb portion is inside of a set of a config such as
<patterndb version="4" pub_date="2009-09-01"> <ruleset name="named" id="some-unique-value"> <pattern>named</pattern> <rules>
[ your included pattern db data ]
</ruleset> </pattendb>
It is the line <pattern>named</pattern> that matches the parsed program name from the syslog line to allow the rule to be parsed against the syslog message.
For testing purposes I make a destination to write a json object. That way I can see everything that populated.
template t_json { template("$(format-json --scope everything )\n"); template_escape(no); }; destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000" template(t_json)); };
Evan.
On 10/27/2017 05:22 AM, Robert Webb wrote:
So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1.
Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng.
I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point.
I have also been told that I may have to do a custom python parser instead.
Any help or guidance is appreciated.
Here is my rule:
<rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system">
<patterns>
<pattern>queries: info: client @IPvANY:named.client.ip@#@ NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern>
</patterns>
<examples>
<example>
<test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message>
<test_values>
<test_value name="named.client.ip">213.46. 246.51</test_value>
<test_value name="named.client.port">1890</test_value>
<test_value name="named.domain.name">tumtali.fegade.com </test_value>
<test_value name="named.record.type">A</test_value>
<test_value name="named.record.flags">-</test_value>
<test_value name="named.server.ip">103.5. 198.211</test_value>
</test_values>
</example>
<example>
<test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message>
<test_values>
<test_value name="named.client.ip">10.187. 195.186</test_value>
<test_value name="named.client.port">40441</test_value>
<test_value name="named.domain.name"> banister.paars.tld</test_value>
<test_value name="named.record.type">A</test_value>
<test_value name="named.record.flags">-EDC</test_value>
<test_value name="named.server.ip">10.226. 134.25</test_value>
</test_values>
</example>
<example>
<test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message>
<test_values>
<test_value name="named.client.ip">192. 168.102.189</test_value>
<test_value name="named.client.port">58454</test_value>
<test_value name="named.domain.name">maps.googleapis.com </test_value>
<test_value name="named.record.type">A</test_value>
<test_value name="named.record.flags">+</test_value>
<test_value name="named.server.ip">192. 168.102.2</test_value>
</test_values>
</example>
</examples>
<values>
</values>
<tags>
<tag>named</tag>
<tag>queries</tag>
</tags>
</rule>
Here are test results:
Pattern matching part:
queries: info: client @IP:named.client.ip=137.75.76.138@ #@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@ (@IP:named.server.ip=137.75.75.75@)
Matching part:
queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)
Values:
MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)
PROGRAM=named
.classifier.class=system
.classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee
named.client.ip=137.75.76.138
named.client.port=38643
named.domain.name=(sr.symcb.com)
named.record.type=A
named.server.ip=137.75.75.75
named.host.name=sr.symcb.com
named.record.flags=+E
TAGS=.classifier.system,named,queries
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq