<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks again Evan, So it should work as expected. <div class=""><br class=""></div><div class="">Could you share how you defined the patterndb in the conf or comment on mine and did you need to write all your own pattern file or were you able to leverage the community ? <div class=""><br class=""></div><div class="">I have a REST api to an ipam server and from there I generate a filter,destination and log for each subnet which is used for log files but my ES destination and log destination is very basic. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""># SYSLOG-NG.CONF</div><div class="">@version:3.8</div><div class="">@include "scl.conf"</div><div class="">@module mod-java</div></div><div class=""><br class=""></div><div class=""><div class="">parser pattern_db {</div><div class=""> db-parser(</div><div class=""> file("/var/lib/syslog-ng/patterndb.xml")</div><div class=""> );</div><div class="">};</div></div><div class=""><div class="">source s_netsyslog {</div><div class=""> udp();</div><div class=""> tcp();</div><div class="">};</div></div><div class=""><div class="">destination d_es {</div><div class=""> elasticsearch2(</div><div class=""> index("syslog-ng_${YEAR}.${MONTH}.${DAY}")</div><div class=""> type("syslog-ng") # Description: The type of the index. For example, type("test")</div><div class=""> port("9300")</div><div class=""> server("127.0.0.1")</div><div class=""> client-mode("transport")</div><div class=""> skip-cluster-health-check("yes")</div><div class=""> cluster("meo")</div><div class=""> resource("/etc/elasticsearch/elasticsearch.yml")</div><div class=""> client_lib_dir("/usr/share/elasticsearch/lib")</div><div class=""> );</div><div class="">};</div></div><div class=""><div class=""><div class="">log {</div><div class=""> source(s_netsyslog);</div><div class=""> parser(pattern_db);</div><div class=""> destination(d_es);</div><div class=""> };</div></div></div><div class=""><div class=""> <br class=""><div><blockquote type="cite" class=""><div class="">On Sep 12, 2016, at 2:25 PM, Evan Rempel <<a href="mailto:erempel@uvic.ca" class="">erempel@uvic.ca</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">Using the patterndb to associate
metadata with each syslog message is great, however, you have to
ensure that those additional fields are output into the json
object that you send to elasticsearch.<br class="">
<br class="">
We do exactly what you are trying to do, so that our elasticsearch
document contains all of the fields parsed by our patterndb.<br class="">
<br class="">
as an example we have a log line of<br class="">
<br class="">
<meta http-equiv="content-type" content="text/html; charset=utf-8" class="">
September 12th 2016, 11:17:25.836 <a href="http://chiru.comp.uvic.ca" class="">chiru.comp.uvic.ca</a>
<a href="http://mail.info" class="">mail.info</a> in.imapproxyd: LOGOUT: '"vgmodi"' from server sd
[200]<br class="">
<br class="">
and an elasticsearch document shown below. The cfgmgr* fields come
from our asset management system. The PATTERNID comes from our
pattern database entry and the user and sd fields come from the
patterndb data parsers.<br class="">
<br class="">
You don't need to have a pattern for every log line at the start.
The second example below is a syslog line that does NOT match any
pattern in our database.<br class="">
<br class="">
{<br class="">
"_index": "flare-2016.09.12.11",<br class="">
"_type": "flare",<br class="">
"_id": "AVcfnhBMpdjtwzWgS7rU",<br class="">
"_score": 1,<br class="">
"_source": {<br class="">
"user": "vgmodi",<br class="">
"sd": "200",<br class="">
"flare": {<br class="">
"profile": "DCS"<br class="">
},<br class="">
"cfgmgrrole": "ADMIN",<br class="">
"cfgmgrosFull": "Redhat 5_64",<br class="">
"cfgmgros": "unix",<br class="">
"cfgmgrmodel": "ESX 5",<br class="">
"cfgmgrlocation": "ESX-BCP",<br class="">
"cfgmgrenvironment": "BCP PROD",<br class="">
"cfgmgrassetType": "Virtual Server",<br class="">
"SOURCEHOST": "<a href="http://chiru.comp.uvic.ca" class="">chiru.comp.uvic.ca</a>",<br class="">
"SHORTHOST": "chiru",<br class="">
"PROGRAM": "in.imapproxyd",<br class="">
"PRIORITY": "info",<br class="">
"PID": "5129",<br class="">
"PATTERNID": "864",<br class="">
"MESSAGE": "LOGOUT: '\"vgmodi\"' from server sd [200]",<br class="">
"ISODATE": "2016-09-12T11:17:25.836-07:00",<br class="">
"HOST": "<a href="http://chiru.comp.uvic.ca" class="">chiru.comp.uvic.ca</a>",<br class="">
"FACILITY": "mail"<br class="">
},<br class="">
"fields": {<br class="">
"ISODATE": [<br class="">
1473704234822<br class="">
]<br class="">
}<br class="">
}<br class="">
<br class="">
Non-matching log line.<br class="">
<br class="">
{<br class="">
"_index": "flare-2016.09.12.11",<br class="">
"_type": "flare",<br class="">
"_id": "AVcfpLpIpdjtwzWgXfVD",<br class="">
"_score": 1,<br class="">
"_source": {<br class="">
"flare": {<br class="">
"profile": "DCS"<br class="">
},<br class="">
"cfgmgrrole": "ADMIN",<br class="">
"cfgmgrosFull": "Redhat 6_64",<br class="">
"cfgmgros": "unix",<br class="">
"cfgmgrmodel": "ESX 5",<br class="">
"cfgmgrlocation": "ESX-PROD",<br class="">
"cfgmgrenvironment": "Prod",<br class="">
"cfgmgrassetType": "Virtual Server",<br class="">
"SOURCEHOST": "<a href="http://tyrant.comp.uvic.ca" class="">tyrant.comp.uvic.ca</a>",<br class="">
"SHORTHOST": "tyrant",<br class="">
"PROGRAM": "cas",<br class="">
"PRIORITY": "info",<br class="">
"MESSAGE": "prod: [ajp-apr-8009-exec-37]: Mon Sep 12 11:24:31
PDT
2016,CAS,SERVICE_TICKET_NOT_CREATED,<a class="moz-txt-link-freetext" href="https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca">https://www.uvic.ca/netlink/j_spring_cas_security_check,audit:unknown,206.87.181.44,www.uvic.ca</a>",<br class="">
"ISODATE": "2016-09-12T11:24:31.000-07:00",<br class="">
"HOST": "<a href="http://tyrant.comp.uvic.ca" class="">tyrant.comp.uvic.ca</a>",<br class="">
"FACILITY": "daemon"<br class="">
},<br class="">
"fields": {<br class="">
"ISODATE": [<br class="">
1473704671000<br class="">
]<br class="">
}<br class="">
}<br class="">
<br class="">
<br class="">
On 09/12/2016 11:08 AM, Scot Needy wrote:<br class="">
</div>
<blockquote cite="mid:7EC0E214-04BA-485A-89C0-3149B50CCF33@gmail.com" type="cite" class="">
<pre wrap="" class="">Hello List,
I’m trying to understand the use case of pattern_db when the destination will be ES. My initial understanding was that I could use patterndb as an engine to tag my log message data with attributes, but it doesn’t seem to work that way. I have a json output like this in Kibana.
In a loghost deployment, It looks like I would need to manually align a patterndb filter with each host_message type even before patterned comes into play.
Q) What is the right solution for enriching message data into ES ?
Example JSON from Kibana MESSAGE is not parsed.
=======================
{
"_index": "syslog-ng_2016.09.12",
"_type": "syslog-ng",
"_id": "AVcdnzJla9VjMdxDYo8Z",
"_score": null,
"_source": {
"PROGRAM": “###-asa11",
"PRIORITY": "warning",
"MESSAGE": "%ASA-4-106023: Deny tcp src outside:###.###.31.2/33553 dst public:###.###.7.191/443 by access-group \"outside_access_in\" [0x2c1c6a65, 0x0]",
"ISODATE": "2016-09-12T13:57:03-04:00",
"HOST": “###.###.###.###",
"FACILITY": "local5",
"@timestamp": "2016-09-12T13:57:03-04:00"
},
"fields": {
"ISODATE": [
1473703023000
],
"@timestamp": [
1473703023000
]
},
"sort": [
1473703023000
]
}
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br class=""><p class=""><br class="">
</p>
<pre class="moz-signature" cols="500">--
Evan Rempel <a class="moz-txt-link-abbreviated" href="mailto:erempel@uvic.ca">erempel@uvic.ca</a>
Senior Systems Administrator 250.721.7691
Data Centre Services, University Systems, University of Victoria
</pre>
</div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></div></div></div></body></html>