[syslog-ng] How to export syslogng to elasticsearch

[Czanik, Péter]

Hi,

You don't seem to have the Java module available. Is it a distribution
package? Those usually don't have it enabled. Check
https://syslog-ng.org/3rd-party-binaries/ for unofficial package for
openSUSE, Fedora, Debian and Ubuntu, which are known to have Java
support enabled. Also check
https://www.balabit.com/blog/troubleshooting-java-support-syslog-ng/
and the documentation.

Bye,
Peter Czanik (CzP) <peter.czanik at balabit.com>
Balabit / syslog-ng upstream
https://www.balabit.com/blog/author/peterczanik/
https://twitter.com/PCzanik


On Thu, May 18, 2017 at 4:44 PM, 'Mik J' via SYSLOG-NG
<syslog-ng at balabit.com> wrote:
> Hello,
>
> I've read many documentation about how to export syslogng to elasticsearch but still don't get it
>
> # syslog-ng -V
> syslog-ng 3.7.2
> Installer-Version: 3.7.2
> Revision:
> Compile-Date: May  8 2017 10:54:55
> Available-Modules: afprog,afsocket,afsql,afuser,basicfuncs,confgen,csvparser,dbparser,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,syslogformat,system-source,affile,cryptofuncs
> Enable-Debug: off
> Enable-GProf: off
> Enable-Memtrace: off
> Enable-IPv6: on
> Enable-Spoof-Source: off
> Enable-TCP-Wrapper: off
> Enable-Linux-Caps: off
>
>
> In syslog-ng.conf I need to add:
>
> @module mod-java
> @include "scl.conf"
>
> In scl.conf I have (I kept the default configuration)
>
> @define scl-root "`syslog-ng-data`/include/scl"
> @define include-path "`include-path`:`syslog-ng-data`/include"
> @include 'scl/*/*.conf'
> I don't understand really what should I configure.
>
> I also downloaded gradle
> # ls /usr/local/gradle-3.5
>
> And Java
> # /usr/local/jdk-1.8.0/bin/java -version
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
>
>
>
> Back to syslog-ng.conf I added
> # Test Elasticsearch
> filter f_MyTest { host("1.1.1.1"); };
> log {
> source(s_net);
> filter(f_MyTest);
> destination(d_MyTest);
> };
>
> destination d_elastic {
>  elasticsearch(
>  index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
>  type("test")
>  cluster("syslog-ng")
>  client_mode("transport")
>  custom_id("${UNIQID}")
>  flush-limit("10000")
>  );
> };
> log {
>  source(s_net);
>  filter(f_MyTest);
>  destination(d_elastic);
>  flags(flow-control);
> };
>
>
> But starting the daemon gives me this error
> # syslog-ng
> [2017-05-18T16:43:42.465496] Plugin module not found in 'module-path'; module-path='/usr/local/lib/syslog-ng', module='mod-java'
> Error parsing destination, destination plugin elasticsearch not found in /etc/syslog-ng/syslog-ng.conf at line 171, column 2:
> elasticsearch(
> ^^^^^^^^^^^^^
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>