[syslog-ng] errors with elasticsearch_http on 2.21 & 3.22

[Russell Fulton]

I have had this config running happily for several months and today I tried to add the geoip2 plugin and managed to break everything.

I first realised that I needed to recompile syslog-ng to get the geoip2 plugin but forgot I was running a locally compiled version .

I tried  apt install syslog-ng-mod-geoip and this broke things badly and I have been trying to recover for the last few hours :(

The issue is that any attempt to start syslog-ng with the original config that includes a destination of elasticsearch_http get an error:

Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.22/etc/syslog-ng.conf:33:3-33:21

Judging from what I found on th ’Net  this is what one would expect if the syslog-ng version was prior to 3.21 and did not support the http dest.

I first tried recompiling and installing 3.21 (in a different dir) with geoip2 enabled.  That went with out errors but still would not process my config.

Aside:  At this point I found that in spite of having the geoip2 module included, syslog-ng failed to find it when I copied the setup from the manual.  long story short the manual says the module is ‘geoip2’ when in fact it is ‘geoip2-plugin’ and that is the name you have to use in the @module.  Weird!

I then compiled 3.22 since I had the tar file and this behaved the same.

rful011 at secmgrprd02:~$ cat short.conf
@version: 3.21
@include "scl.conf"

@module geoip2-plugin

source s_loghost {
    network( transport("tcp") flags(no-multi-line) port(1514) keep-alive(yes));
};


destination d_elastic {
  elasticsearch_http(
    index("auth_${YEAR}.${MONTH}.${DAY}")
   type("auth")
   persist-name("auth")
    template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
    url("http://secesprd01.its.auckland.ac.nz:9200/_bulk")
  );
};

log {
    source(s_loghost);
    destination(d_elastic );
    flags(flow-control);
};

rful011 at secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng -s -c ~/short.conf
Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.21/etc/syslog-ng.conf:33:3-33:21:
28                  file( "/home/rful011/test.log" template(t_db_parsed_test));
29       };
30
31
32      destination d_elastic {
33---->   elasticsearch_http(
33---->   ^^^^^^^^^^^^^^^^^^
34          index("auth_${YEAR}.${MONTH}.${DAY}")
35         type("auth")
36         persist-name("auth")
37          template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
38          url("http://secesprd01.its.auckland.ac.nz:9200/_bulk”)

rful011 at secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng --version
syslog-ng 3 (3.21.1)
Config version: 3.21
Installer-Version: 3.21.1
Revision:
Compile-Date: Mar 29 2020 11:57:39
Module-Directory: /usr/local/syslog-ng-3.21/lib/syslog-ng
Module-Path: /usr/local/syslog-ng-3.21/lib/syslog-ng
Include-Path: /usr/local/syslog-ng-3.21/share/syslog-ng/include
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip2-plugin,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off
Enable-Systemd: off



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200329/3b8490c2/attachment-0001.html>