<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">That sounds like a pretty cool setup. Wish I had the time but my "we" is a mirror. &nbsp;<div class=""><div class=""><br class=""></div><div class="">Guess your doing the routing with filter definitions then?&nbsp;<br class=""><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 12, 2016, at 4:19 PM, Evan Rempel &lt;<a href="mailto:erempel@uvic.ca" class="">erempel@uvic.ca</a>&gt; wrote:</div><br class="Apple-interchange-newline"><div class="">
  
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
  
  <div bgcolor="#FFFFFF" text="#000000" class="">
    <div class="moz-cite-prefix">It is not so much the patterndb, but
      the name value pairs that you place into the json object. We use a
      template of<br class="">
      <br class="">
      template( $(format-json --scope rfc5424,nv-pairs \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude DATE \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude 0 \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude 1 \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude 2 \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude LEGACY_MSGHDR \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude AUTHPROGRAM \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude SOURCE \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude HOST_FROM \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude synopsis \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude allofit \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude therest \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude R_* \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --exclude S_* \<br class="">
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --key ISODATE )\n")<br class="">
      <br class="">
      Some of the exclusions are due to values that our patterndb
      populates. All of the names that are specific to our database
      patterns are:<br class="">
      DATE, 0, 1, 2, AUTHPROGRAM, synopsis, allofit, therest<br class="">
      <br class="">
      <br class="">
      We did write all of our own database patterns. We wrote our own
      web front end to a database and auto-generate the syslog-ng
      pattern database. Just a little over 5,000 patterns and growing.<br class="">
      <br class="">
      We took a different approach. Rather than rewriting the syslog-ng
      config, filtering and logging details, we classify messages with a
      tag from the patterndb. A static syslog-ng configuration that
      matches on the tags from the patterndb and filters and logs based
      on the pattern tags.<br class="">
      <br class="">
      We route messages to different analysis engines based on the
      message classification. Soome backends are:<br class="">
      <br class="">
      - create an alert/ticket<br class="">
      - rate threshold trigger of and event<br class="">
      - collect multiple lines together and alert a digest of the log
      lines<br class="">
      - execute an external program<br class="">
      - start/continue timer - heartbeat events<br class="">
      <br class="">
      There are lots of others that are on our roadmap.<br class="">
      <br class="">
      I hope this description gets you started.<br class="">
      <br class="">
      Evan.<br class="">
      <br class="">
      <br class="">
      On 09/12/2016 12:29 PM, Scot Needy wrote:<br class="">
    </div>
    <blockquote cite="mid:848F0274-6F07-47B5-A814-BD75EAA508AB@gmail.com" type="cite" class="">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
      Thanks again Evan, So it should work as expected.&nbsp;
      <div class=""><br class="">
      </div>
      <div class="">Could you share how you defined the patterndb in the
        conf or comment on mine and did you need to write all your own
        pattern file or were you able to leverage the community ?&nbsp;
        <div class=""><br class="">
        </div>
        <div class="">I have a REST api to an ipam server and from there
          I generate a filter,destination and log for each subnet which
          is used for log files but my ES destination and log
          destination is very basic. &nbsp;</div>
        <div class=""><br class="">
        </div>
        <div class=""><br class="">
        </div>
        <div class="">
          <div class=""># SYSLOG-NG.CONF</div>
          <div class="">@version:3.8</div>
          <div class="">@include "scl.conf"</div>
          <div class="">@module mod-java</div>
        </div>
        <div class=""><br class="">
        </div>
        <div class="">
          <div class="">parser pattern_db {</div>
          <div class="">&nbsp; db-parser(</div>
          <div class="">&nbsp; &nbsp; file("/var/lib/syslog-ng/patterndb.xml")</div>
          <div class="">&nbsp; );</div>
          <div class="">};</div>
        </div>
        <div class="">
          <div class="">source s_netsyslog {</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; udp();</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; tcp();</div>
          <div class="">};</div>
        </div>
        <div class="">
          <div class="">destination d_es {</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; elasticsearch2(</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
            index("syslog-ng_${YEAR}.${MONTH}.${DAY}")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; type("syslog-ng") # Description:
            The type of the index. For example, type("test")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; port("9300")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; server("127.0.0.1")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; client-mode("transport")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; skip-cluster-health-check("yes")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; cluster("meo")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
            resource("/etc/elasticsearch/elasticsearch.yml")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
            client_lib_dir("/usr/share/elasticsearch/lib")</div>
          <div class="">&nbsp; &nbsp; &nbsp; &nbsp; );</div>
          <div class="">};</div>
        </div>
        <div class="">
          <div class="">
            <div class="">log {</div>
            <div class="">&nbsp; &nbsp; &nbsp; &nbsp; source(s_netsyslog);</div>
            <div class="">&nbsp; &nbsp; &nbsp; &nbsp; parser(pattern_db);</div>
            <div class="">&nbsp; &nbsp; &nbsp; &nbsp; destination(d_es);</div>
            <div class="">&nbsp;};</div>
          </div>
        </div>
        <div class="">
          <div class="">&nbsp;<br class="">
            <div class="">
              <blockquote type="cite" class="">
                <div class="">On Sep 12, 2016, at 2:25 PM, Evan Rempel
                  &lt;<a moz-do-not-send="true" href="mailto:erempel@uvic.ca" class="">erempel@uvic.ca</a>&gt;
                  wrote:</div>
                <br class="Apple-interchange-newline">
                <div class="">
                  <div bgcolor="#FFFFFF" text="#000000" class="">
                    <div class="moz-cite-prefix">Using the patterndb to
                      associate metadata with each syslog message is
                      great, however, you have to ensure that those
                      additional fields are output into the json object
                      that you send to elasticsearch.<br class="">
                      <br class="">
                      We do exactly what you are trying to do, so that
                      our elasticsearch document contains all of the
                      fields parsed by our patterndb.<br class="">
                      <br class="">
                      as an example we have a log line of<br class="">
                      <br class="">
                      September 12th 2016, 11:17:25.836&nbsp;&nbsp;&nbsp; <a moz-do-not-send="true" href="http://chiru.comp.uvic.ca/" class="">chiru.comp.uvic.ca</a>&nbsp;&nbsp;&nbsp;
                      <a moz-do-not-send="true" href="http://mail.info/" class="">mail.info</a>&nbsp;&nbsp;&nbsp; in.imapproxyd: LOGOUT:
                      '"vgmodi"' from server sd [200]<br class="">
                      <br class="">
                      and an elasticsearch document shown below. The
                      cfgmgr* fields come from our asset management
                      system. The PATTERNID comes from our pattern
                      database entry and the user and sd fields come
                      from the patterndb data parsers.<br class="">
                      <br class="">
                      You don't need to have a pattern for every log
                      line at the start. The second example below is a
                      syslog line that does NOT match any pattern in our
                      database.<br class="">
                      <br class="">
                      {<br class="">
                      &nbsp; "_index": "flare-2016.09.12.11",<br class="">
                      &nbsp; "_type": "flare",<br class="">
                      &nbsp; "_id": "AVcfnhBMpdjtwzWgS7rU",<br class="">
                      &nbsp; "_score": 1,<br class="">
                      &nbsp; "_source": {<br class="">
                      &nbsp;&nbsp;&nbsp; "user": "vgmodi",<br class="">
                      &nbsp;&nbsp;&nbsp; "sd": "200",<br class="">
                      &nbsp;&nbsp;&nbsp; "flare": {<br class="">
                      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "profile": "DCS"<br class="">
                      &nbsp;&nbsp;&nbsp; },<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrrole": "ADMIN",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrosFull": "Redhat 5_64",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgros": "unix",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrmodel": "ESX 5",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrlocation": "ESX-BCP",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrenvironment": "BCP PROD",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrassetType": "Virtual Server",<br class="">
                      &nbsp;&nbsp;&nbsp; "SOURCEHOST": "<a moz-do-not-send="true" href="http://chiru.comp.uvic.ca/" class="">chiru.comp.uvic.ca</a>",<br class="">
                      &nbsp;&nbsp;&nbsp; "SHORTHOST": "chiru",<br class="">
                      &nbsp;&nbsp;&nbsp; "PROGRAM": "in.imapproxyd",<br class="">
                      &nbsp;&nbsp;&nbsp; "PRIORITY": "info",<br class="">
                      &nbsp;&nbsp;&nbsp; "PID": "5129",<br class="">
                      &nbsp;&nbsp;&nbsp; "PATTERNID": "864",<br class="">
                      &nbsp;&nbsp;&nbsp; "MESSAGE": "LOGOUT: '\"vgmodi\"' from server
                      sd [200]",<br class="">
                      &nbsp;&nbsp;&nbsp; "ISODATE": "2016-09-12T11:17:25.836-07:00",<br class="">
                      &nbsp;&nbsp;&nbsp; "HOST": "<a moz-do-not-send="true" href="http://chiru.comp.uvic.ca/" class="">chiru.comp.uvic.ca</a>",<br class="">
                      &nbsp;&nbsp;&nbsp; "FACILITY": "mail"<br class="">
                      &nbsp; },<br class="">
                      &nbsp; "fields": {<br class="">
                      &nbsp;&nbsp;&nbsp; "ISODATE": [<br class="">
                      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1473704234822<br class="">
                      &nbsp;&nbsp;&nbsp; ]<br class="">
                      &nbsp; }<br class="">
                      }<br class="">
                      <br class="">
                      Non-matching log line.<br class="">
                      <br class="">
                      {<br class="">
                      &nbsp; "_index": "flare-2016.09.12.11",<br class="">
                      &nbsp; "_type": "flare",<br class="">
                      &nbsp; "_id": "AVcfpLpIpdjtwzWgXfVD",<br class="">
                      &nbsp; "_score": 1,<br class="">
                      &nbsp; "_source": {<br class="">
                      &nbsp;&nbsp;&nbsp; "flare": {<br class="">
                      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "profile": "DCS"<br class="">
                      &nbsp;&nbsp;&nbsp; },<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrrole": "ADMIN",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrosFull": "Redhat 6_64",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgros": "unix",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrmodel": "ESX 5",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrlocation": "ESX-PROD",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrenvironment": "Prod",<br class="">
                      &nbsp;&nbsp;&nbsp; "cfgmgrassetType": "Virtual Server",<br class="">
                      &nbsp;&nbsp;&nbsp; "SOURCEHOST": "<a moz-do-not-send="true" href="http://tyrant.comp.uvic.ca/" class="">tyrant.comp.uvic.ca</a>",<br class="">
                      &nbsp;&nbsp;&nbsp; "SHORTHOST": "tyrant",<br class="">
                      &nbsp;&nbsp;&nbsp; "PROGRAM": "cas",<br class="">
                      &nbsp;&nbsp;&nbsp; "PRIORITY": "info",<br class="">
                      &nbsp;&nbsp;&nbsp; "MESSAGE": "prod: [ajp-apr-8009-exec-37]: Mon
                      Sep 12 11:24:31 PDT
                      2016,CAS,SERVICE_TICKET_NOT_CREATED,<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca"></a><a class="moz-txt-link-freetext" href="https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca">https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca</a>",<br class="">
                      &nbsp;&nbsp;&nbsp; "ISODATE": "2016-09-12T11:24:31.000-07:00",<br class="">
                      &nbsp;&nbsp;&nbsp; "HOST": "<a moz-do-not-send="true" href="http://tyrant.comp.uvic.ca/" class="">tyrant.comp.uvic.ca</a>",<br class="">
                      &nbsp;&nbsp;&nbsp; "FACILITY": "daemon"<br class="">
                      &nbsp; },<br class="">
                      &nbsp; "fields": {<br class="">
                      &nbsp;&nbsp;&nbsp; "ISODATE": [<br class="">
                      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1473704671000<br class="">
                      &nbsp;&nbsp;&nbsp; ]<br class="">
                      &nbsp; }<br class="">
                      }<br class="">
                      <br class="">
                      <br class="">
                      On 09/12/2016 11:08 AM, Scot Needy wrote:<br class="">
                    </div>
                    <blockquote cite="mid:7EC0E214-04BA-485A-89C0-3149B50CCF33@gmail.com" type="cite" class="">
                      <pre class="" wrap="">Hello List, 

  
I’m trying to understand the use case of pattern_db when the destination will be ES. My initial understanding was that I could use patterndb as an engine to tag my log message data with attributes, but it doesn’t seem to work that way. I have a json output like this in Kibana. 

In a loghost deployment, It looks like I would need to manually align a patterndb filter with each host_message type even before patterned comes into play.  

Q) What is the right solution for enriching message data into ES ? 

Example JSON from Kibana MESSAGE is not parsed.  
=======================
{
  "_index": "syslog-ng_2016.09.12",
  "_type": "syslog-ng",
  "_id": "AVcdnzJla9VjMdxDYo8Z",
  "_score": null,
  "_source": {
    "PROGRAM": “###-asa11",
    "PRIORITY": "warning",
    "MESSAGE": "%ASA-4-106023: Deny tcp src outside:###.###.31.2/33553 dst public:###.###.7.191/443 by access-group \"outside_access_in\" [0x2c1c6a65, 0x0]",
    "ISODATE": "2016-09-12T13:57:03-04:00",
    "HOST": “###.###.###.###",
    "FACILITY": "local5",
    "@timestamp": "2016-09-12T13:57:03-04:00"
  },
  "fields": {
    "ISODATE": [
      1473703023000
    ],
    "@timestamp": [
      1473703023000
    ]
  },
  "sort": [
    1473703023000
  ]
}


</pre>
                    </blockquote>
                  </div>
                </div>
              </blockquote>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br class="">
  </div>

______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></div></div></div></body></html>