<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">The program name will have an implied
@ANYSTRING@ on the end, so if the syslog payload could<br>
be parsed to detect the program name of %ASA..... then your
patterndb.xml would work.<br>
The problem is that they syslog payload can not be parsed.<br>
<br>
What we do is use a hand crafted patterndb to detect all of the
different problem formats<br>
that Cisco logs (in our environment) and change<br>
<br>
a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the
XXX is the leading characters of the %XXX-#-##### of the syslog
body.<br>
b) the MESSAGE to be %XXX-#-####... for the rest of the line.<br>
<br>
All of the poor hosts, sequence numbers, timestamps etc are all
thrown away.<br>
<br>
After that, the internal buffers of syslog-ng contain a usable
PROGRAM and MESSAGE such that we can use<br>
a patterndb to match the message part of the log line.<br>
<br>
This does mean that we have two patterndb parsers for every log
line but it seems to work well for us.<br>
What would work better is if Cisco would fix their logging, but
that isn't going to happen in my lifetime :-(<br>
<br>
I hope that makes sense.<br>
<br>
Evan.<br>
<br>
On 02/27/2018 05:37 PM, Tim Ghetti wrote:<br>
</div>
<blockquote type="cite"
cite="mid:a9736d3b780548db91ba3884ff910b4e@targetedsupport.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Ok, that makes
sense. I sort of suspected that this was due to the program
name not matching, since pdbtool works when specifying the
program name. Do you know if there is a way to configure
patterndb so that it matches the program name, regardless of
the full program name with event code? Is it possible to
regex the program or use the patterndb format i.e.
<pattern>%ASA@ANYSTRING::@</pattern> or
something similar?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span
style="color:windowtext"> syslog-ng
[<a class="moz-txt-link-freetext" href="mailto:syslog-ng-bounces@lists.balabit.hu">mailto:syslog-ng-bounces@lists.balabit.hu</a>]
<b>On Behalf Of </b>Evan Rempel<br>
<b>Sent:</b> Tuesday, February 27, 2018 8:26 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><br>
<b>Subject:</b> Re: [syslog-ng] Cisco ASA parsing with
patterndb/elasticsearch<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Welcome to the horrible world of Cisco
logging :-(<br>
<br>
The issue you are bumping into is that when these log lines
are parsed by syslog-ng (or any<br>
syslog daemon that does not specifically understand Cisco
logs) there is no program name, or the<br>
program name is the full %ASA-4-106023 part of the log line.
There are so many ways that Cisco<br>
can log incorrectly it is not possible to say without seeing
your exact log lines.<br>
<br>
Cisco can add a * to the date/time stamp to indicate that
there is no time server configured on the device.<br>
That makes the date/time invalid as far as parsing is
concerned.<br>
<br>
Cisco can add a period (.) to the date/time stamp to
indicate that there is a time server configured on<br>
the device, but the time server can not be reached. Again,
this makes the date/time invalid as far as parsing<br>
is concerned.<br>
<br>
Cisco can add a sequence number at the start of the log line
rather than starting the line with a date/6time stamp.<br>
Invalid parsing again.<br>
<br>
Cisco can leave out the sequence number but still include
the trailing colon from the sequence number.
<br>
<br>
You get the idea. So many ways to get it wrong, and they
never get it right :-(<br>
<br>
Evan.<br>
<br>
On 02/27/2018 04:46 PM, Tim Ghetti wrote:<span
style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi – having some trouble getting paterndb
functional and looking for some help. I would like to use
patterndb to parse my cisco ass firewall logs before sending
it to elasticsearch. However when the messages get to
elasticsearch, I don’t see the messages being parsed.
Running pdbtool against the logs seems to work.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"># pdbtool match -p
/etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f
/var/log/asatest.log |more<o:p></o:p></p>
<p class="MsoNormal">HOST=<span style="color:#1F497D">X.X.X.X</span><o:p></o:p></p>
<p class="MsoNormal">MESSAGE=Built dynamic TCP translation
from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X<o:p></o:p></p>
<p class="MsoNormal">PROGRAM=%ASA-6-305011<o:p></o:p></p>
<p class="MsoNormal">LEGACY_MSGHDR=%ASA-6-305011: <o:p></o:p></p>
<p class="MsoNormal">.classifier.class=system<o:p></o:p></p>
<p class="MsoNormal">.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b<o:p></o:p></p>
<p class="MsoNormal">Protocol=TCP<o:p></o:p></p>
<p class="MsoNormal">GlobalIP=X.X.X.X<o:p></o:p></p>
<p class="MsoNormal">GlobalPort=X<o:p></o:p></p>
<p class="MsoNormal">LocalIP=X.X.X.X<o:p></o:p></p>
<p class="MsoNormal">LocalPort=X<o:p></o:p></p>
<p class="MsoNormal">TAGS=.classifier.system<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">**********************<o:p></o:p></p>
<p class="MsoNormal">SYSLOG-NG CONF FILE<o:p></o:p></p>
<p class="MsoNormal">@version: 3.11<o:p></o:p></p>
<p class="MsoNormal">source s_network { tcp(); udp(); };<o:p></o:p></p>
<p class="MsoNormal">destination d_elastic {<o:p></o:p></p>
<p class="MsoNormal"> elasticsearch2(<o:p></o:p></p>
<p class="MsoNormal"> client-mode("http")<o:p></o:p></p>
<p class="MsoNormal"> cluster("ITESCL001")<o:p></o:p></p>
<p class="MsoNormal">
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")<o:p></o:p></p>
<p class="MsoNormal"> cluster-url("<a
href="http://X.X.X.X:9200" moz-do-not-send="true">http://X.X.X.X:9200</a>")<o:p></o:p></p>
<p class="MsoNormal"> type("syslog")<o:p></o:p></p>
<p class="MsoNormal"> flush-limit("1")<o:p></o:p></p>
<p class="MsoNormal"> );<o:p></o:p></p>
<p class="MsoNormal">};<o:p></o:p></p>
<p class="MsoNormal">destination d_catchall {
file("/var/log/catchall.log"); };<o:p></o:p></p>
<p class="MsoNormal">filter f_ciscoasa { host("X.X.X.X"); };<o:p></o:p></p>
<p class="MsoNormal">parser p_ciscoasa
{db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};<o:p></o:p></p>
<p class="MsoNormal">log { source(s_network);
filter(f_ciscoasa); parser(p_ciscoasa);
destination(d_elastic); flags(final, flow-control); };<o:p></o:p></p>
<p class="MsoNormal">log { source(s_network);
destination(d_catchall); };<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">**********************<o:p></o:p></p>
<p class="MsoNormal">PATTERNDB FILE<o:p></o:p></p>
<p class="MsoNormal"><?xml version='1.0'
encoding='UTF-8'?><o:p></o:p></p>
<p class="MsoNormal"><patterndb version='4'
pub_date='2018-02-19'><o:p></o:p></p>
<p class="MsoNormal"> <ruleset name='%ASA'
id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'><o:p></o:p></p>
<p class="MsoNormal"> <pattern>%ASA</pattern><o:p></o:p></p>
<p class="MsoNormal"> <description><o:p></o:p></p>
<p class="MsoNormal"> This ruleset covers the Cisco ASA
firewalls<o:p></o:p></p>
<p class="MsoNormal"> </description><o:p></o:p></p>
<p class="MsoNormal"> <rules><o:p></o:p></p>
<p class="MsoNormal"> <rule provider="%ASA"
id="b3de7699-8213-c744-944e-9413298afe86" class="system"><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 1594 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Teardown
@ESTRING:Protocol: @connection for faddr
@IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
@IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Teardown ICMP connection for faddr
X.X.X.X/X gaddr X.X.X.X/X laddr
X.X.X.X/X</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule
id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system'
provider='%ASA'><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 1369 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Teardown dynamic
@ESTRING:Protocol: @translation from
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to
@ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:
@duration@ANYSTRING::@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Teardown dynamic UDP translation from
any:X.X.X.X/X to outside:X.X.X.X/X duration
0:00:00</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule
id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system'
provider='%ASA'><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 1254 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Teardown
@ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort:
@@ESTRING::@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Teardown UDP connection 55101037 for
outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes
132</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule
id='00c0732d-1e34-7340-a75f-21198bf71137' class='system'
provider='%ASA'><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 1256 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Built outbound
@ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@
to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort:
@(@ESTRING::)@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Built outbound UDP connection 55101037 for
outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X
(X.X.X.X/X)</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule
id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system'
provider='%ASA'><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 1594 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Built inbound
@ESTRING:Protocol: @connection for faddr
@IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
@IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Built inbound ICMP connection for faddr
X.X.X.X/X gaddr X.X.X.X/X laddr
X.X.X.X/X</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule
id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system'
provider='%ASA'><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 1763 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Built inbound
@ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to
identity:@IPv4:DstIP:/@@ESTRING:DstPort:
@(@ESTRING::)@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Built inbound UDP connection 55101078 for
inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X
(X.X.X.X/X)</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule
id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system'
provider='%ASA'><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 1539 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Teardown
@ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to
identity:@IPv4:DstIP:/@@ESTRING:DstPort:
@duration@ANYSTRING::@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Teardown UDP connection 55101084 for
inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01
bytes 88</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule
id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system'
provider='%ASA'><o:p></o:p></p>
<p class="MsoNormal"> <!-- support: 3648 --><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Built dynamic
@ESTRING:Protocol: @translation from
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to
@ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> <examples><o:p></o:p></p>
<p class="MsoNormal"> <example><o:p></o:p></p>
<p class="MsoNormal"> <test_message
program='%ASA'>Built dynamic TCP translation from
any:X.X.X.X/X to outside:X.X.X.X/X</test_message><o:p></o:p></p>
<p class="MsoNormal"> </example><o:p></o:p></p>
<p class="MsoNormal"> </examples><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> <rule provider='%ASA'
class='system' id='39'><o:p></o:p></p>
<p class="MsoNormal"> <patterns><o:p></o:p></p>
<p class="MsoNormal"> <pattern>Cleared
@ESTRING:: @urgent flag from
@ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING:
::@@ESTRING::/@@NUMBER::@</pattern><o:p></o:p></p>
<p class="MsoNormal"> <pattern>regular
translation creation failed for @ESTRING:: @src
@ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type
@NUMBER::@, code @NUMBER::@</pattern><o:p></o:p></p>
<p class="MsoNormal"> </patterns><o:p></o:p></p>
<p class="MsoNormal"> </rule><o:p></o:p></p>
<p class="MsoNormal"> </rules><o:p></o:p></p>
<p class="MsoNormal"> </ruleset><o:p></o:p></p>
<p class="MsoNormal"></patterndb><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">N�n�r����)em�h�yhiם�w^��</pre>
</blockquote>
<p><br>
</p>
</body>
</html>