<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">Circling back around to this question and another log entry having issues.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">So I have some custom logging happening on a server using tcpdump. Basically, tcpdump is running and when a hit takes place, the data is sent to syslog via the logger command. This part is working and the
entries are making it to syslog-ng.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">I have used pdbtool to test my xml file and all works. I even verified that the tag is correct so the pattern will match.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">What I have a problem with is in testing with pdbtool, if I use the standard double quote, I get the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">pdbtool match -D -c -p /etc/syslog-ng/patterndb.d/sinkhole.xml -P "sinkhole" -M " www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co.
A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.xxx.xxx.xxx.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1
(192)"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">-bash: !]: event not found<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">However, if I use single quotes, I get what I expect:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">pdbtool match -D -c -p /etc/syslog-ng/patterndb.d/sinkhole.xml -P "sinkhole" -M ' www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co.
A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1
(192)'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Pattern matching part:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">@ESTRING:None= www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 @ @HOSTNAME:sinkhole.domain.name=cadhya.co.@@ANYSTRING:None=
NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)@<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Matching part:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"> www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co. NS
140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Values:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">MESSAGE= www.example.com.domain > 137.xxx.xxx.xxx.47177: [bad udp cksum 0xab79 -> 0x6ff0!] 52861* q: A? www.cadhya.co. 1/2/4 www.cadhya.co. A 127.0.0.1 ns: cadhya.co. NS 137.xxx.xxx.xxx.cadhya.co., cadhya.co.
NS 140.172.111.111.cadhya.co. ar: 137.xxx.xxx.xxx.cadhya.co. A 127.0.0.1, 137.xxx.xxx.xxx.cadhya.co. AAAA ::1, 140.172.111.111.cadhya.co. A 127.0.0.1, 140.172.111.111.cadhya.co. AAAA ::1 (192)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">PROGRAM=sinkhole<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">.classifier.class=system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">.classifier.rule_id=21071031002<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">sinkhole.domain.name=cadhya.co.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">TAGS=.classifier.system,sinkhole,queries<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Robert Webb
<br>
<b>Sent:</b> Friday, October 27, 2017 10:56 AM<br>
<b>To:</b> Evan Rempel <erempel@uvic.ca><br>
<b>Cc:</b> syslog-ng@lists.balabit.hu<br>
<b>Subject:</b> RE: [syslog-ng] Need to parse named query log entries and get just the domain portion<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Evan,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Thanks you for pointing me in the right direction. I was so focused on the matching starting @ “queries: info:” that I totally forgot that the first part of the actual MSG has the date and time sent with it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">My pattern now takes that into account.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Message: 27-Oct-2017 14:19:42.359 queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><pattern>@ESTRING:: @@ESTRING:: @queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@
@ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Robert<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Evan Rempel [<a href="mailto:erempel@uvic.ca">mailto:erempel@uvic.ca</a>]
<br>
<b>Sent:</b> Friday, October 27, 2017 9:52 AM<br>
<b>To:</b> Robert Webb <<a href="mailto:rwebb@ropeguru.com">rwebb@ropeguru.com</a>><br>
<b>Subject:</b> Re: [syslog-ng] Need to parse named query log entries and get just the domain portion<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">In the past when I have had this problem it was in the patterndb portion just outside if what you have included.<br>
<br>
essentially the program does not match, so the rule is never attempted. This results in none of the tag-value pairs being populated. The pdbtool shows a match because the pattern is always in the correct program.<br>
<br>
Your patterndb portion is inside of a set of a config such as<br>
<br>
<patterndb version="4" pub_date="2009-09-01"><br>
<ruleset name="named" id="some-unique-value"><br>
<pattern>named</pattern><br>
<rules><br>
<br>
[ your included pattern db data ]<br>
<br>
</ruleset><br>
</pattendb><br>
<br>
It is the line<br>
<pattern>named</pattern><br>
that matches the parsed program name from the syslog line to allow the rule to be parsed against the<br>
syslog message.<br>
<br>
For testing purposes I make a destination to write a json object. That way I can see everything that populated.<br>
<br>
template t_json { template("$(format-json --scope everything )\n"); template_escape(no); };<br>
destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000" template(t_json)); };<br>
<br>
Evan.<br>
<br>
On 10/27/2017 05:22 AM, Robert Webb wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do
all this inside syslog-ng.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue
after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I have also been told that I may have to do a custom python parser instead.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Any help or guidance is appreciated.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Here is my rule:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message><o:p></o:p></p>
<p class="MsoNormal"> <test_values><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.client.ip">213.46.246.51</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.client.port">1890</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.domain.name">tumtali.fegade.com</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.record.type">A</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.record.flags">-</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.server.ip">103.5.198.211</test_value><o:p></o:p></p>
<p class="MsoNormal"> </test_values><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message><o:p></o:p></p>
<p class="MsoNormal"> <test_values><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.client.ip">10.187.195.186</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.client.port">40441</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.domain.name">banister.paars.tld</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.record.type">A</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.record.flags">-EDC</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.server.ip">10.226.134.25</test_value><o:p></o:p></p>
<p class="MsoNormal"> </test_values><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message><o:p></o:p></p>
<p class="MsoNormal"> <test_values><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.client.ip">192.168.102.189</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.client.port">58454</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.domain.name">maps.googleapis.com</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.record.type">A</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.record.flags">+</test_value><o:p></o:p></p>
<p class="MsoNormal"> <test_value name="named.server.ip">192.168.102.2</test_value><o:p></o:p></p>
<p class="MsoNormal"> </test_values><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> <values><o:p></o:p></p>
<p class="MsoNormal"> </values><o:p></o:p></p>
<p class="MsoNormal"> <tags><o:p></o:p></p>
<p class="MsoNormal"> <tag>named</tag><o:p></o:p></p>
<p class="MsoNormal"> <tag>queries</tag><o:p></o:p></p>
<p class="MsoNormal"> </tags><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Here are test results:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Pattern matching part:<o:p></o:p></p>
<p class="MsoNormal">queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@)<o:p></o:p></p>
<p class="MsoNormal">Matching part:<o:p></o:p></p>
<p class="MsoNormal">queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)<o:p></o:p></p>
<p class="MsoNormal">Values:<o:p></o:p></p>
<p class="MsoNormal">MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com): query: sr.symcb.com IN A +E (137.75.75.75)<o:p></o:p></p>
<p class="MsoNormal">PROGRAM=named<o:p></o:p></p>
<p class="MsoNormal">.classifier.class=system<o:p></o:p></p>
<p class="MsoNormal">.classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee<o:p></o:p></p>
<p class="MsoNormal">named.client.ip=137.75.76.138<o:p></o:p></p>
<p class="MsoNormal">named.client.port=38643<o:p></o:p></p>
<p class="MsoNormal">named.domain.name=(sr.symcb.com)<o:p></o:p></p>
<p class="MsoNormal">named.record.type=A<o:p></o:p></p>
<p class="MsoNormal">named.server.ip=137.75.75.75<o:p></o:p></p>
<p class="MsoNormal">named.host.name=sr.symcb.com<o:p></o:p></p>
<p class="MsoNormal">named.record.flags=+E<o:p></o:p></p>
<p class="MsoNormal">TAGS=.classifier.system,named,queries<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>