[syslog-ng] attempting to get parsed logs into elasticsearch

30 Sep 2016

      Hi folks

I am trying to get some parsed logs into elasticssearch but have ended up with a cryptic error message:

Starting syslog-ng
/usr/lib/jvm/java-1.7.0-oracle-1.7.0.91.x86_64/jre/lib/amd64/server
[2016-09-30T12:43:43.649899] Error initializing message pipeline;

which almost certainly relates to the ES set up but I have no idea what is actually wrong.  The ES logs do not show anything.

Config file:
@version: 3.8
@module mod-java
@include "scl.conf"

options {
        use_dns (no);
        use_fqdn (no);
        keep_hostname (yes);
};

source s_loghost {
    tcp(flags(no-multi-line) port(1514) keep-alive(yes));
};

destination d_syslog { file("/var/log/syslog.log"); };

destination d_elastic {
  elasticsearch(
    index("auth_${YEAR}.${MONTH}.${DAY}")
    type("auth")
    cluster("security")
    flush-limit("1000")
  );
};

parser p_patterns {  db-parser( file("/etc/syslog-ng/merged.xml")); };

log {

    source(s_loghost);
    parser (p_patterns);
    destination(d_elastic );

};

The same configuration with a json file destination works fine.

Any hints on what to look at appreciated.

The ES instance running on the host is set to data: no and I expect it to ship the data to one of the other nodes which has storage.

Russell (who admits to being an ES novice)

[syslog-ng] attempting to get parsed logs into elasticsearch

Russell Fulton