[syslog-ng] Help with ES destination error

25 Jan 2018

      Hi all,
Hope you can help me solve this scenario.
I am receiving messages from a firewall, extracting KV pairs with a custom parser_db, sending them to an elasticsearch destination applying a JSON template.

All works fine until I apply the JSON template to the Elasticsearch destination - when I do this in the config and a message is received I get the following from Syslog:

syslog-ng    | [2018-01-25T17:59:00.873601] Outgoing message; message='{"timestamp":"2018-01-25 17:59:00","src":{"port":"62118","ip":"192.168.xx.xx","if":"X1","hostname":"hostname.domain.co.uk"},"sn":"xxxx","sid":"5165","priority":"1","nipspri":"3","msg":"IPS Detection Alert","ipscat":"WEB-TLS SSLv2.0 Client Hello 2","fw":{"ip":"x.xxx.xxx.xx","action":"NA"},"dst":{"port":"443","ip":"40.xxx.xxx.xx","if":"X5"},"_classifier":{"rule_id":"71593655-6fa7-4fca-9617-480e79703215","class":"IPSDetection"},"PROGRAM":"id=Firewall","LEGACY_MSGHDR":"id=Firewall ","HOST_FROM":"xx.xx.xx.xx","HOST":"xx.xx.xx.xx"}\x0a’

syslog-ng    | 17:59:00.897 [?3?] ERROR  - {"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}}

My setup is all on Ubuntu/Docker and is using Syslog-ng 3.13.2 and ES 6.1.

Configuration is quite simple:

source s_net {
  udp(
    ip(0.0.0.0),port(514) #,flags(no-parse)
  );
};

#this parse all the field from the previous message
parser sonicwall {
  db-parser(file("/etc/syslog-ng/patterndb.d/sonicwall-pattern.xml"));
};

#some more parsing due to firewall sending dynamic lenght messages for sources and destinations
parser split-sonicwall-srcdata {
  csv-parser(
    columns("src.ip", "src.port", "src.if","src.hostname")
    delimiters(chars(":"), strings(" "))
    template("${src.data}")
  );
};

parser split-sonicwall-dstdata {
  csv-parser(
    columns("dst.ip", "dst.port", "dst.if","dst.hostname")
    delimiters(chars(":"), strings(" "))
    template("${dst.data}")
  );
};

#finally the template applied

template t_sonicwall {
    template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,mfield,cfield,nfield)\n");
};

destination d_elasticsearch {
  elasticsearch2(
    client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/")
    index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')")
    type("syslog")
    client_mode("http")
    cluster("docker-cluster")
    cluster_url("http://elasticsearch:9200")
    template(t_sonicwall)
    flush-limit("1")
  );
};

log {
  source(s_net);
  parser(sonicwall);
  parser(split-sonicwall-srcdata);
  parser(split-sonicwall-dstdata);
  destination(d_elasticsearch);
  destination(d_file);
  };

Although if I try to manually put the content of the message with a curl POST request it works (not some info is masked with xx):

curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{
"timestamp": "2018-01-25 17:59:00",
"src.port": "62118",
"src.ip": "xx.xx.xx.xx",
"src.if": "X1",
"src.hostname": "hostname.domain.co.uk",
"sn": “xxxx",
"sid": "5165",
"priority": "1",
"nipspri": "3",
"msg": "IPS Detection Alert",
"ipscat": "WEB-TLS SSLv2.0 Client Hello 2",
"fw.ip": "xx.xx.xx.xx",
"fw.action": "NA",
"dst.port": "443",
"dst.ip": "xx.xx.xx.xx",
"dst.if": "X5",
"_classifier.rule_id": "71593655-6fa7-4fca-9617-480e79703215",
"_classifier.class": "IPSDetection",
"PROGRAM": “id=Firewall",
"LEGACY_MSGHDR": "id=Firewall ",
"HOST_FROM": "xx.xx.xx.xx",
"HOST": "xx.xx.xx.xx"
}' "http://es:9200/test/test/“

{"_index":"test","_type":"test","_id":"ZzaCLmEB5A1B2qoXkU_p","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}

Any suggestions would be much appreciated.

Thanks,
Marco Mignone