[syslog-ng] [FORGED] errors with elasticsearch_http on 2.21 & 3.22 -- correction
Antal Nemes (anemes)
Antal.Nemes at oneidentity.com
Mon Mar 30 06:48:09 UTC 2020
Hello,
I quickly checked 3.21, and elasticsearch-http with compiled source and it works for me. I attached my console log. It might help if you compare the commands with yours.
Just couple of ideas that might have went wrong.
When you tried to use from package: the only dependency of elasticsearch-http is the http module. You need to install syslog-ng-mod-http, and not syslog-ng-mod-geoip.
Depending where you get the packages: but in 3.22, there were two geoip modules: geoip and geoip2. For example if you use OBS, you would have syslog-ng-mod-geoip and syslog-ng-mod-geoip2. You need to install syslog-ng-mod-geoip2 to use geip2.
In 3.26, however, geoip was removed, and only geoip2 remains. The driver was also renamed, but as I see the package remains geoip2. On the other hand, there is still a syslog-ng-mod-geoip package, looking for the original .so file. I do not understand how that could be generated. I will look into that. But I do not think this would be a problem for you.
When you compiled from source. Did you install a curl development package? That is necessary for the http module to be compiled. Unless otherwise specified, there is an autodetection, and if libcurl was not found, http module is simply not built. You can force availability by adding `--enable-http` during configure, similarly to that I did in the attached log. At the end of the configure, you need to see yes for http module.
[...]
HTTP support (module) : yes
[...]
Was the installation directory clean before you make installed? Is it possible syslog-ng modules could be installed there from a different syslog-ng version? Leftover or injected so-s, might be picked up by syslog-ng, even if they are not from the correct version. That can cause runtime problems.
Br,
Antal
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Russell Fulton <r.fulton at auckland.ac.nz>
Sent: Sunday, March 29, 2020 21:23
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] [FORGED] errors with elasticsearch_http on 2.21 & 3.22 -- correction
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
small correction to the test invocation of syslog-ng below. should be:
rful011 at secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng -s -f ~/short.conf
rather than -c. It does not effect the error messages…. It was running on the full config not the stripped down one.
R
On 29/03/2020, at 4:03 PM, Russell Fulton <r.fulton at auckland.ac.nz<mailto:r.fulton at auckland.ac.nz>> wrote:
I have had this config running happily for several months and today I tried to add the geoip2 plugin and managed to break everything.
I first realised that I needed to recompile syslog-ng to get the geoip2 plugin but forgot I was running a locally compiled version .
I tried apt install syslog-ng-mod-geoip and this broke things badly and I have been trying to recover for the last few hours :(
The issue is that any attempt to start syslog-ng with the original config that includes a destination of elasticsearch_http get an error:
Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.22/etc/syslog-ng.conf:33:3-33:21
Judging from what I found on th ’Net this is what one would expect if the syslog-ng version was prior to 3.21 and did not support the http dest.
I first tried recompiling and installing 3.21 (in a different dir) with geoip2 enabled. That went with out errors but still would not process my config.
Aside: At this point I found that in spite of having the geoip2 module included, syslog-ng failed to find it when I copied the setup from the manual. long story short the manual says the module is ‘geoip2’ when in fact it is ‘geoip2-plugin’ and that is the name you have to use in the @module. Weird!
I then compiled 3.22 since I had the tar file and this behaved the same.
rful011 at secmgrprd02:~$ cat short.conf
@version: 3.21
@include "scl.conf"
@module geoip2-plugin
source s_loghost {
network( transport("tcp") flags(no-multi-line) port(1514) keep-alive(yes));
};
destination d_elastic {
elasticsearch_http(
index("auth_${YEAR}.${MONTH}.${DAY}")
type("auth")
persist-name("auth")
template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
url("http://secesprd01.its.auckland.ac.nz:9200/_bulk<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsecesprd01.its.auckland.ac.nz%3A9200%2F_bulk&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C79eabe24c46e40c8e4d108d7d416a7ad%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637211066100185486&sdata=zq1UUYGy3IE2TF4Lb%2FEwiWWlbZDw5GrYbsuWB2Pe1e0%3D&reserved=0>")
);
};
log {
source(s_loghost);
destination(d_elastic );
flags(flow-control);
};
rful011 at secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng -s -c ~/short.conf
Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.21/etc/syslog-ng.conf:33:3-33:21:
28 file( "/home/rful011/test.log" template(t_db_parsed_test));
29 };
30
31
32 destination d_elastic {
33----> elasticsearch_http(
33----> ^^^^^^^^^^^^^^^^^^
34 index("auth_${YEAR}.${MONTH}.${DAY}")
35 type("auth")
36 persist-name("auth")
37 template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
38 url("http://secesprd01.its.auckland.ac.nz:9200/_bulk”)
rful011 at secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng --version
syslog-ng 3 (3.21.1)
Config version: 3.21
Installer-Version: 3.21.1
Revision:
Compile-Date: Mar 29 2020 11:57:39
Module-Directory: /usr/local/syslog-ng-3.21/lib/syslog-ng
Module-Path: /usr/local/syslog-ng-3.21/lib/syslog-ng
Include-Path: /usr/local/syslog-ng-3.21/share/syslog-ng/include
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip2-plugin,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off
Enable-Systemd: off
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200330/ea402ae8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: syslog-ng.3.21-console.log
Type: text/x-log
Size: 36518 bytes
Desc: syslog-ng.3.21-console.log
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200330/ea402ae8/attachment-0001.bin>
More information about the syslog-ng
mailing list