Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

28 Feb 2018

      The program name will have an implied @ANYSTRING@ on the end, so if the 
syslog payload could
be parsed to detect the program name of %ASA..... then your 
patterndb.xml would work.
The problem is that they syslog payload can not be parsed.

What we do is use a hand crafted patterndb to detect all of the 
different problem formats
that Cisco logs (in our environment) and change

a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is 
the leading characters of the %XXX-#-##### of the syslog body.
b) the MESSAGE to be %XXX-#-####... for the rest of the line.

All of the poor hosts, sequence numbers, timestamps etc are all thrown away.

After that, the internal buffers of syslog-ng contain a usable PROGRAM 
and MESSAGE such that we can use
a patterndb to match the message part of the log line.

This does mean that we have two patterndb parsers for every log line but 
it seems to work well for us.
What would work better is if Cisco would fix their logging, but that 
isn't going to happen in my lifetime :-(

I hope that makes sense.

Evan.

On 02/27/2018 05:37 PM, Tim Ghetti wrote:
...
Ok, that makes sense. I sort of suspected that this was due to the 
program name not matching, since pdbtool works when specifying the 
program name. Do you know if there is a way to configure patterndb so 
that it matches the program name, regardless of the full program name 
with event code? Is it possible to regex the program or use the 
patterndb format i.e. <pattern>%ASA@ANYSTRING::@</pattern> or 
something similar?
Tim
*From:*syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On 
Behalf Of *Evan Rempel
*Sent:* Tuesday, February 27, 2018 8:26 PM
*To:* syslog-ng@lists.balabit.hu
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch
Welcome to the horrible world of Cisco logging :-(
The issue you are bumping into is that when these log lines are parsed 
by syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there 
is no program name, or the
program name is the full %ASA-4-106023 part of the log line. There are 
so many ways that Cisco
can log incorrectly it is not possible to say without seeing your 
exact log lines.
Cisco can add a * to the date/time stamp to indicate that there is no 
time server configured on the device.
That makes the date/time invalid as far as parsing is concerned.
Cisco can add a period (.) to the date/time stamp to indicate that 
there is a time server configured on
the device, but the time server can not be reached. Again, this makes 
the date/time invalid as far as parsing
is concerned.
Cisco can add a sequence number at the start of the log line rather 
than starting the line with a date/6time stamp.
Invalid parsing again.
Cisco can leave out the sequence number but still include the trailing 
colon from the sequence number.
You get the idea. So many ways to get it wrong, and they never get it 
right :-(
Evan.
On 02/27/2018 04:46 PM, Tim Ghetti wrote:
Hi – having some trouble getting paterndb functional and looking
    for some help. I would like to use patterndb to parse my cisco ass
    firewall logs before sending it to elasticsearch. However when the
    messages get to elasticsearch, I don’t see the messages being
    parsed. Running  pdbtool against the logs seems to work.
# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA
    -f /var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to
    OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
LEGACY_MSGHDR=%ASA-6-305011:
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system
**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
    elasticsearch2(
        client-mode("http")
        cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
        cluster-url("http://X.X.X.X:9200")
        type("syslog")
        flush-limit("1")
    );
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa
    {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa);
    destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };
**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
  <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
  <pattern>%ASA</pattern>
  <description>
    This ruleset covers the Cisco ASA firewalls
  </description>
    <rules>
      <rule provider="%ASA"
    id="b3de7699-8213-c744-944e-9413298afe86" class="system">
        <!-- support: 1594 -->
        <patterns>
          <pattern>Teardown @ESTRING:Protocol: @connection for
    faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr
    @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
    @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown ICMP
    connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr
    X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6'
    class='system' provider='%ASA'>
        <!-- support: 1369 -->
        <patterns>
          <pattern>Teardown dynamic @ESTRING:Protocol:
    @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort:
    @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:
    @duration@ANYSTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown dynamic UDP
    translation from any:X.X.X.X/X to outside:X.X.X.X/X duration
    0:00:00</test_message>
            </example>
        </examples>
      </rule>
      <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87'
    class='system' provider='%ASA'>
        <!-- support: 1254 -->
        <patterns>
          <pattern>Teardown @ESTRING:Protocol: @connection
    @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to
    @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown UDP
    connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X
    duration 0:00:00 bytes 132</test_message>
            </example>
        </examples>
      </rule>
      <rule id='00c0732d-1e34-7340-a75f-21198bf71137'
    class='system' provider='%ASA'>
        <!-- support: 1256 -->
        <patterns>
          <pattern>Built outbound @ESTRING:Protocol: @connection
    @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort:
    @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort:
    @(@ESTRING::)@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built outbound UDP
    connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to
    inside:X.X.X.X/X (X.X.X.X/X)</test_message>
            </example>
        </examples>
      </rule>
      <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d'
    class='system' provider='%ASA'>
        <!-- support: 1594 -->
        <patterns>
          <pattern>Built inbound @ESTRING:Protocol: @connection
    for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr
    @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
    @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built inbound ICMP
    connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr
    X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e'
    class='system' provider='%ASA'>
        <!-- support: 1763 -->
        <patterns>
          <pattern>Built inbound @ESTRING:Protocol: @connection
    @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort:
    @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort:
    @(@ESTRING::)@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built inbound UDP
    connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to
    identity:X.X.X.X/X (X.X.X.X/X)</test_message>
            </example>
        </examples>
      </rule>
      <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21'
    class='system' provider='%ASA'>
        <!-- support: 1539 -->
        <patterns>
          <pattern>Teardown @ESTRING:Protocol: @connection
    @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to
    identity:@IPv4:DstIP:/@@ESTRING:DstPort:
    @duration@ANYSTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown UDP
    connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X
    duration 0:02:01 bytes 88</test_message>
            </example>
        </examples>
      </rule>
      <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b'
    class='system' provider='%ASA'>
        <!-- support: 3648 -->
        <patterns>
          <pattern>Built dynamic @ESTRING:Protocol: @translation
    from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to
    @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built dynamic TCP
    translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule provider='%ASA' class='system' id='39'>
        <patterns>
          <pattern>Cleared @ESTRING:: @urgent flag from
    @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING:
    ::@@ESTRING::/@@NUMBER::@</pattern>
          <pattern>regular translation creation failed for
    @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING:
    ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern>
        </patterns>
      </rule>
    </rules>
  </ruleset>
</patterndb>
N�n�r����)em�h�yhiם�w^��