<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi all,<div class="">Hope you can help me solve this scenario.</div><div class="">I am receiving messages from a firewall, extracting KV pairs with a custom parser_db, sending them to an elasticsearch destination applying a JSON template.</div><div class=""><br class=""></div><div class="">All works fine until I apply the JSON template to the Elasticsearch destination - when I do this in the config and a message is received I get the following from Syslog:</div><div class=""><br class=""></div><div class=""><div class="">syslog-ng | [2018-01-25T17:59:00.873601] Outgoing message; message='{"timestamp":"2018-01-25 17:59:00","src":{"port":"62118","ip":"192.168.xx.xx","if":"X1","hostname":"<a href="http://hostname.domain.co.uk" class="">hostname.domain.co.uk</a>"},"sn":"xxxx","sid":"5165","priority":"1","nipspri":"3","msg":"IPS Detection Alert","ipscat":"WEB-TLS SSLv2.0 Client Hello 2","fw":{"ip":"x.xxx.xxx.xx","action":"NA"},"dst":{"port":"443","ip":"40.xxx.xxx.xx","if":"X5"},"_classifier":{"rule_id":"71593655-6fa7-4fca-9617-480e79703215","class":"IPSDetection"},"PROGRAM":"id=Firewall","LEGACY_MSGHDR":"id=Firewall ","HOST_FROM":"xx.xx.xx.xx","HOST":"xx.xx.xx.xx"}\x0a’</div><div class=""><br class=""></div><div class="">syslog-ng | 17:59:00.897 [?3?] ERROR - {"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}}</div></div><div class=""><br class=""></div><div class="">My setup is all on Ubuntu/Docker and is using <b class="">Syslog-ng 3.13.2</b> and <b class="">ES 6.1</b>.</div><div class=""><br class=""></div><div class="">Configuration is quite simple:</div><div class=""><br class=""></div><div class=""><div class="">source s_net {</div><div class=""> udp(</div><div class=""> ip(0.0.0.0),port(514) #,flags(no-parse)</div><div class=""> );</div><div class="">};</div></div><div class=""><br class=""></div><div class="">#this parse all the field from the previous message</div><div class=""><div class="">parser sonicwall {</div><div class=""> db-parser(file("/etc/syslog-ng/patterndb.d/sonicwall-pattern.xml"));</div><div class="">};</div><div class=""><br class=""></div><div class="">#some more parsing due to firewall sending dynamic lenght messages for sources and destinations</div><div class="">parser split-sonicwall-srcdata {</div><div class=""> csv-parser(</div><div class=""> columns("src.ip", "src.port", "src.if","src.hostname")</div><div class=""> delimiters(chars(":"), strings(" "))</div><div class=""> template("${src.data}")</div><div class=""> );</div><div class="">};</div><div class=""><br class=""></div><div class="">parser split-sonicwall-dstdata {</div><div class=""> csv-parser(</div><div class=""> columns("dst.ip", "dst.port", "dst.if","dst.hostname")</div><div class=""> delimiters(chars(":"), strings(" "))</div><div class=""> template("${dst.data}")</div><div class=""> );</div><div class="">};</div></div><div class=""><br class=""></div><div class="">#finally the template applied</div><div class=""><br class=""></div><div class=""><div class=""><div class="">template t_sonicwall {</div><div class=""> template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,mfield,cfield,nfield)\n");</div><div class="">};</div></div></div><div class=""><br class=""></div><div class=""><div class="">destination d_elasticsearch {</div><div class=""> elasticsearch2(</div><div class=""> client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/")</div><div class=""> index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')")</div><div class=""> type("syslog")</div><div class=""> client_mode("http")</div><div class=""> cluster("docker-cluster")</div><div class=""> cluster_url("<a href="http://elasticsearch:9200" class="">http://elasticsearch:9200</a>")</div><div class=""> template(t_sonicwall)</div><div class=""> flush-limit("1")</div><div class=""> );</div><div class="">};</div></div><div class=""><br class=""></div><div class=""><div class="">log {</div><div class=""> source(s_net);</div><div class=""> parser(sonicwall);</div><div class=""> parser(split-sonicwall-srcdata);</div><div class=""> parser(split-sonicwall-dstdata);</div><div class=""> destination(d_elasticsearch);</div><div class=""> destination(d_file);</div><div class=""> };</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Although if I try to manually put the content of the message with a curl POST request it works (not some info is masked with xx):</div><div class=""><br class=""></div><div class=""><div class="">curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{</div><div class="">"timestamp": "2018-01-25 17:59:00",</div><div class="">"src.port": "62118",</div><div class="">"src.ip": "xx.xx.xx.xx",</div><div class="">"src.if": "X1",</div><div class="">"src.hostname": "<a href="http://hostname.domain.co.uk" class="">hostname.domain.co.uk</a>",</div><div class="">"sn": “xxxx",</div><div class="">"sid": "5165",</div><div class="">"priority": "1",</div><div class="">"nipspri": "3",</div><div class="">"msg": "IPS Detection Alert",</div><div class="">"ipscat": "WEB-TLS SSLv2.0 Client Hello 2",</div><div class="">"fw.ip": "xx.xx.xx.xx",</div><div class="">"fw.action": "NA",</div><div class="">"dst.port": "443",</div><div class="">"dst.ip": "xx.xx.xx.xx",</div><div class="">"dst.if": "X5",</div><div class="">"_classifier.rule_id": "71593655-6fa7-4fca-9617-480e79703215",</div><div class="">"_classifier.class": "IPSDetection",</div><div class="">"PROGRAM": “id=Firewall",</div><div class="">"LEGACY_MSGHDR": "id=Firewall ",</div><div class="">"HOST_FROM": "xx.xx.xx.xx",</div><div class="">"HOST": "xx.xx.xx.xx"</div><div class="">}' "<a href="http://es:9200/test/test" class="">http://es:9200/test/test</a>/“</div></div><div class=""><br class=""></div><div class="">{"_index":"test","_type":"test","_id":"ZzaCLmEB5A1B2qoXkU_p","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Any suggestions would be much appreciated.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Marco Mignone</div></div></body></html>