Hi all, Hope you can help me solve this scenario. I am receiving messages from a firewall, extracting KV pairs with a custom parser_db, sending them to an elasticsearch destination applying a JSON template. All works fine until I apply the JSON template to the Elasticsearch destination - when I do this in the config and a message is received I get the following from Syslog: syslog-ng | [2018-01-25T17:59:00.873601] Outgoing message; message='{"timestamp":"2018-01-25 17:59:00","src":{"port":"62118","ip":"192.168.xx.xx","if":"X1","hostname":"hostname.domain.co.uk"},"sn":"xxxx","sid":"5165","priority":"1","nipspri":"3","msg":"IPS Detection Alert","ipscat":"WEB-TLS SSLv2.0 Client Hello 2","fw":{"ip":"x.xxx.xxx.xx","action":"NA"},"dst":{"port":"443","ip":"40.xxx.xxx.xx","if":"X5"},"_classifier":{"rule_id":"71593655-6fa7-4fca-9617-480e79703215","class":"IPSDetection"},"PROGRAM":"id=Firewall","LEGACY_MSGHDR":"id=Firewall ","HOST_FROM":"xx.xx.xx.xx","HOST":"xx.xx.xx.xx"}\x0a’ syslog-ng | 17:59:00.897 [?3?] ERROR - {"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}} My setup is all on Ubuntu/Docker and is using Syslog-ng 3.13.2 and ES 6.1. Configuration is quite simple: source s_net { udp( ip(0.0.0.0),port(514) #,flags(no-parse) ); }; #this parse all the field from the previous message parser sonicwall { db-parser(file("/etc/syslog-ng/patterndb.d/sonicwall-pattern.xml")); }; #some more parsing due to firewall sending dynamic lenght messages for sources and destinations parser split-sonicwall-srcdata { csv-parser( columns("src.ip", "src.port", "src.if","src.hostname") delimiters(chars(":"), strings(" ")) template("${src.data}") ); }; parser split-sonicwall-dstdata { csv-parser( columns("dst.ip", "dst.port", "dst.if","dst.hostname") delimiters(chars(":"), strings(" ")) template("${dst.data}") ); }; #finally the template applied template t_sonicwall { template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,mfield,cfield,nfield)\n"); }; destination d_elasticsearch { elasticsearch2( client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/") index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')") type("syslog") client_mode("http") cluster("docker-cluster") cluster_url("http://elasticsearch:9200") template(t_sonicwall) flush-limit("1") ); }; log { source(s_net); parser(sonicwall); parser(split-sonicwall-srcdata); parser(split-sonicwall-dstdata); destination(d_elasticsearch); destination(d_file); }; Although if I try to manually put the content of the message with a curl POST request it works (not some info is masked with xx): curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{ "timestamp": "2018-01-25 17:59:00", "src.port": "62118", "src.ip": "xx.xx.xx.xx", "src.if": "X1", "src.hostname": "hostname.domain.co.uk", "sn": “xxxx", "sid": "5165", "priority": "1", "nipspri": "3", "msg": "IPS Detection Alert", "ipscat": "WEB-TLS SSLv2.0 Client Hello 2", "fw.ip": "xx.xx.xx.xx", "fw.action": "NA", "dst.port": "443", "dst.ip": "xx.xx.xx.xx", "dst.if": "X5", "_classifier.rule_id": "71593655-6fa7-4fca-9617-480e79703215", "_classifier.class": "IPSDetection", "PROGRAM": “id=Firewall", "LEGACY_MSGHDR": "id=Firewall ", "HOST_FROM": "xx.xx.xx.xx", "HOST": "xx.xx.xx.xx" }' "http://es:9200/test/test/“ {"_index":"test","_type":"test","_id":"ZzaCLmEB5A1B2qoXkU_p","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1} Any suggestions would be much appreciated. Thanks, Marco Mignone
Hi, Your mapping probably differs between 'fw-*' and 'test'. Can you change your syslogng config to index to 'test' instead? cheers
Hi Fabien, Thanks for the reply. I have tried removing all ES data and to let ES re-create the index as brand new with the first request coming from syslog-ng but the error still persists. From the documentation in ES both Index auto-creation and auto-mapping is allowed by default. Do you have any other suggestions? Marco
On 26 Jan 2018, at 09:48, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
Your mapping probably differs between 'fw-*' and 'test'. Can you change your syslogng config to index to 'test' instead?
cheers
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Fabien Wernli
-
Marco Mignone