<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Your guess is correct. The routing is
based on tag filters and feeds into these multiple backend
programs to "do the right thing".<br>
<br>
Evan.<br>
<br>
On 09/12/2016 02:05 PM, Scot Needy wrote:<br>
</div>
<blockquote
cite="mid:51D28007-65BE-488C-AC95-863C021BBB61@gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
That sounds like a pretty cool setup. Wish I had the time but my
"we" is a mirror.
<div class="">
<div class=""><br class="">
</div>
<div class="">Guess your doing the routing with filter
definitions then? <br class="">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 12, 2016, at 4:19 PM, Evan Rempel
<<a moz-do-not-send="true"
href="mailto:erempel@uvic.ca" class="">erempel@uvic.ca</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">It is not so much the
patterndb, but the name value pairs that you place
into the json object. We use a template of<br
class="">
<br class="">
template( $(format-json --scope rfc5424,nv-pairs \<br
class="">
--exclude DATE \<br
class="">
--exclude 0 \<br class="">
--exclude 1 \<br class="">
--exclude 2 \<br class="">
--exclude LEGACY_MSGHDR \<br
class="">
--exclude AUTHPROGRAM \<br
class="">
--exclude SOURCE \<br
class="">
--exclude HOST_FROM \<br
class="">
--exclude synopsis \<br
class="">
--exclude allofit \<br
class="">
--exclude therest \<br
class="">
--exclude R_* \<br
class="">
--exclude S_* \<br
class="">
--key ISODATE )\n")<br
class="">
<br class="">
Some of the exclusions are due to values that our
patterndb populates. All of the names that are
specific to our database patterns are:<br class="">
DATE, 0, 1, 2, AUTHPROGRAM, synopsis, allofit,
therest<br class="">
<br class="">
<br class="">
We did write all of our own database patterns. We
wrote our own web front end to a database and
auto-generate the syslog-ng pattern database. Just
a little over 5,000 patterns and growing.<br
class="">
<br class="">
We took a different approach. Rather than
rewriting the syslog-ng config, filtering and
logging details, we classify messages with a tag
from the patterndb. A static syslog-ng
configuration that matches on the tags from the
patterndb and filters and logs based on the
pattern tags.<br class="">
<br class="">
We route messages to different analysis engines
based on the message classification. Soome
backends are:<br class="">
<br class="">
- create an alert/ticket<br class="">
- rate threshold trigger of and event<br class="">
- collect multiple lines together and alert a
digest of the log lines<br class="">
- execute an external program<br class="">
- start/continue timer - heartbeat events<br
class="">
<br class="">
There are lots of others that are on our roadmap.<br
class="">
<br class="">
I hope this description gets you started.<br
class="">
<br class="">
Evan.<br class="">
<br class="">
<br class="">
On 09/12/2016 12:29 PM, Scot Needy wrote:<br
class="">
</div>
<blockquote
cite="mid:848F0274-6F07-47B5-A814-BD75EAA508AB@gmail.com"
type="cite" class=""> Thanks again Evan, So it
should work as expected.
<div class=""><br class="">
</div>
<div class="">Could you share how you defined the
patterndb in the conf or comment on mine and did
you need to write all your own pattern file or
were you able to leverage the community ?
<div class=""><br class="">
</div>
<div class="">I have a REST api to an ipam
server and from there I generate a
filter,destination and log for each subnet
which is used for log files but my ES
destination and log destination is very basic.
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""># SYSLOG-NG.CONF</div>
<div class="">@version:3.8</div>
<div class="">@include "scl.conf"</div>
<div class="">@module mod-java</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">parser pattern_db {</div>
<div class=""> db-parser(</div>
<div class="">
file("/var/lib/syslog-ng/patterndb.xml")</div>
<div class=""> );</div>
<div class="">};</div>
</div>
<div class="">
<div class="">source s_netsyslog {</div>
<div class=""> udp();</div>
<div class=""> tcp();</div>
<div class="">};</div>
</div>
<div class="">
<div class="">destination d_es {</div>
<div class=""> elasticsearch2(</div>
<div class="">
index("syslog-ng_${YEAR}.${MONTH}.${DAY}")</div>
<div class="">
type("syslog-ng") # Description: The type of
the index. For example, type("test")</div>
<div class=""> port("9300")</div>
<div class="">
server("127.0.0.1")</div>
<div class="">
client-mode("transport")</div>
<div class="">
skip-cluster-health-check("yes")</div>
<div class=""> cluster("meo")</div>
<div class="">
resource("/etc/elasticsearch/elasticsearch.yml")</div>
<div class="">
client_lib_dir("/usr/share/elasticsearch/lib")</div>
<div class=""> );</div>
<div class="">};</div>
</div>
<div class="">
<div class="">
<div class="">log {</div>
<div class=""> source(s_netsyslog);</div>
<div class=""> parser(pattern_db);</div>
<div class=""> destination(d_es);</div>
<div class=""> };</div>
</div>
</div>
<div class="">
<div class=""> <br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Sep 12, 2016, at 2:25
PM, Evan Rempel <<a
moz-do-not-send="true"
href="mailto:erempel@uvic.ca"
class=""><a class="moz-txt-link-abbreviated" href="mailto:erempel@uvic.ca">erempel@uvic.ca</a></a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class="">
<div class="moz-cite-prefix">Using
the patterndb to associate
metadata with each syslog message
is great, however, you have to
ensure that those additional
fields are output into the json
object that you send to
elasticsearch.<br class="">
<br class="">
We do exactly what you are trying
to do, so that our elasticsearch
document contains all of the
fields parsed by our patterndb.<br
class="">
<br class="">
as an example we have a log line
of<br class="">
<br class="">
September 12th 2016,
11:17:25.836 <a
moz-do-not-send="true"
href="http://chiru.comp.uvic.ca/"
class="">chiru.comp.uvic.ca</a>
<a moz-do-not-send="true"
href="http://mail.info/"
class="">mail.info</a>
in.imapproxyd: LOGOUT: '"vgmodi"'
from server sd [200]<br class="">
<br class="">
and an elasticsearch document
shown below. The cfgmgr* fields
come from our asset management
system. The PATTERNID comes from
our pattern database entry and the
user and sd fields come from the
patterndb data parsers.<br
class="">
<br class="">
You don't need to have a pattern
for every log line at the start.
The second example below is a
syslog line that does NOT match
any pattern in our database.<br
class="">
<br class="">
{<br class="">
"_index": "flare-2016.09.12.11",<br
class="">
"_type": "flare",<br class="">
"_id": "AVcfnhBMpdjtwzWgS7rU",<br
class="">
"_score": 1,<br class="">
"_source": {<br class="">
"user": "vgmodi",<br class="">
"sd": "200",<br class="">
"flare": {<br class="">
"profile": "DCS"<br class="">
},<br class="">
"cfgmgrrole": "ADMIN",<br
class="">
"cfgmgrosFull": "Redhat 5_64",<br
class="">
"cfgmgros": "unix",<br
class="">
"cfgmgrmodel": "ESX 5",<br
class="">
"cfgmgrlocation": "ESX-BCP",<br
class="">
"cfgmgrenvironment": "BCP
PROD",<br class="">
"cfgmgrassetType": "Virtual
Server",<br class="">
"SOURCEHOST": "<a
moz-do-not-send="true"
href="http://chiru.comp.uvic.ca/"
class="">chiru.comp.uvic.ca</a>",<br
class="">
"SHORTHOST": "chiru",<br
class="">
"PROGRAM": "in.imapproxyd",<br
class="">
"PRIORITY": "info",<br
class="">
"PID": "5129",<br class="">
"PATTERNID": "864",<br
class="">
"MESSAGE": "LOGOUT:
'\"vgmodi\"' from server sd
[200]",<br class="">
"ISODATE":
"2016-09-12T11:17:25.836-07:00",<br
class="">
"HOST": "<a
moz-do-not-send="true"
href="http://chiru.comp.uvic.ca/"
class="">chiru.comp.uvic.ca</a>",<br
class="">
"FACILITY": "mail"<br class="">
},<br class="">
"fields": {<br class="">
"ISODATE": [<br class="">
1473704234822<br class="">
]<br class="">
}<br class="">
}<br class="">
<br class="">
Non-matching log line.<br class="">
<br class="">
{<br class="">
"_index": "flare-2016.09.12.11",<br
class="">
"_type": "flare",<br class="">
"_id": "AVcfpLpIpdjtwzWgXfVD",<br
class="">
"_score": 1,<br class="">
"_source": {<br class="">
"flare": {<br class="">
"profile": "DCS"<br class="">
},<br class="">
"cfgmgrrole": "ADMIN",<br
class="">
"cfgmgrosFull": "Redhat 6_64",<br
class="">
"cfgmgros": "unix",<br
class="">
"cfgmgrmodel": "ESX 5",<br
class="">
"cfgmgrlocation": "ESX-PROD",<br
class="">
"cfgmgrenvironment": "Prod",<br
class="">
"cfgmgrassetType": "Virtual
Server",<br class="">
"SOURCEHOST": "<a
moz-do-not-send="true"
href="http://tyrant.comp.uvic.ca/"
class="">tyrant.comp.uvic.ca</a>",<br
class="">
"SHORTHOST": "tyrant",<br
class="">
"PROGRAM": "cas",<br class="">
"PRIORITY": "info",<br
class="">
"MESSAGE": "prod:
[ajp-apr-8009-exec-37]: Mon Sep 12
11:24:31 PDT
2016,CAS,SERVICE_TICKET_NOT_CREATED,<a
moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca"><a class="moz-txt-link-freetext" href="https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca">https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca</a></a>",<br
class="">
"ISODATE":
"2016-09-12T11:24:31.000-07:00",<br
class="">
"HOST": "<a
moz-do-not-send="true"
href="http://tyrant.comp.uvic.ca/"
class="">tyrant.comp.uvic.ca</a>",<br
class="">
"FACILITY": "daemon"<br
class="">
},<br class="">
"fields": {<br class="">
"ISODATE": [<br class="">
1473704671000<br class="">
]<br class="">
}<br class="">
}<br class="">
<br class="">
<br class="">
On 09/12/2016 11:08 AM, Scot Needy
wrote:<br class="">
</div>
<blockquote
cite="mid:7EC0E214-04BA-485A-89C0-3149B50CCF33@gmail.com"
type="cite" class="">
<pre class="" wrap="">Hello List,
I’m trying to understand the use case of pattern_db when the destination will be ES. My initial understanding was that I could use patterndb as an engine to tag my log message data with attributes, but it doesn’t seem to work that way. I have a json output like this in Kibana.
In a loghost deployment, It looks like I would need to manually align a patterndb filter with each host_message type even before patterned comes into play.
Q) What is the right solution for enriching message data into ES ?
Example JSON from Kibana MESSAGE is not parsed.
=======================
{
"_index": "syslog-ng_2016.09.12",
"_type": "syslog-ng",
"_id": "AVcdnzJla9VjMdxDYo8Z",
"_score": null,
"_source": {
"PROGRAM": “###-asa11",
"PRIORITY": "warning",
"MESSAGE": "%ASA-4-106023: Deny tcp src outside:###.###.31.2/33553 dst public:###.###.7.191/443 by access-group \"outside_access_in\" [0x2c1c6a65, 0x0]",
"ISODATE": "2016-09-12T13:57:03-04:00",
"HOST": “###.###.###.###",
"FACILITY": "local5",
"@timestamp": "2016-09-12T13:57:03-04:00"
},
"fields": {
"ISODATE": [
1473703023000
],
"@timestamp": [
1473703023000
]
},
"sort": [
1473703023000
]
}
</pre>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
<br class="">
</div>
______________________________________________________________________________<br
class="">
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br
class="">
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br
class="">
FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br
class="">
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
<p><br>
</p>
<pre class="moz-signature" cols="500">--
Evan Rempel <a class="moz-txt-link-abbreviated" href="mailto:erempel@uvic.ca">erempel@uvic.ca</a>
Senior Systems Administrator 250.721.7691
Data Centre Services, University Systems, University of Victoria
</pre>
</body>
</html>