<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
A quick correction. After double checking the geoip related removal stuff: it seems fine. I misread the package list. Actually, the last generated syslog-ng-mod-geoip was in 3.23.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<a href="http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04/amd64/" id="LPlnk289271">http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04/amd64/</a></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In 3.26, we only have syslog-ng-mod-geoip2.<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Antal Nemes (anemes) <Antal.Nemes@oneidentity.com><br>
<b>Sent:</b> Monday, March 30, 2020 8:48<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] [FORGED] errors with elasticsearch_http on 2.21 & 3.22 -- correction</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div>
<div>
<div>
<div dir="ltr">
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Hello,</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
I quickly checked 3.21, and elasticsearch-http with compiled source and it works for me. I attached my console log. It might help if you compare the commands with yours.<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Just couple of ideas that might have went wrong.<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
When you tried to use from package: the only dependency of elasticsearch-http is the http module. You need to install syslog-ng-mod-http, and not syslog-ng-mod-geoip.</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Depending where you get the packages: but in 3.22, there were two geoip modules: geoip and geoip2. For example if you use OBS, you would have syslog-ng-mod-geoip and syslog-ng-mod-geoip2. You need to install syslog-ng-mod-geoip2 to use geip2.</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
In 3.26, however, geoip was removed, and only geoip2 remains. The driver was also renamed, but as I see the package remains geoip2. On the other hand, there is still a syslog-ng-mod-geoip package, looking for the original .so file. I do not understand how that
could be generated. I will look into that. But I do not think this would be a problem for you.</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
When you compiled from source. Did you install a curl development package? That is necessary for the http module to be compiled. Unless otherwise specified, there is an autodetection, and if libcurl was not found, http module is simply not built. You can force
availability by adding `--enable-http` during configure, similarly to that I did in the attached log. At the end of the configure, you need to see yes for http module.<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
[...]<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<span> HTTP support (module) : yes<br>
</span><span></span>[...]<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Was the installation directory clean before you make installed? Is it possible syslog-ng modules could be installed there from a different syslog-ng version? Leftover or injected so-s, might be picked up by syslog-ng, even if they are not from the correct version.
That can cause runtime problems.<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
<br>
</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Br,</div>
<div style="color:black; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Antal<br>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Russell Fulton <r.fulton@auckland.ac.nz><br>
<b>Sent:</b> Sunday, March 29, 2020 21:23<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] [FORGED] errors with elasticsearch_http on 2.21 & 3.22 -- correction</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>small correction to the test invocation of syslog-ng below. should be:
<div class="">
<div class=""><br class="">
</div>
<div class=""><font face="Menlo" class="">rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng -s
<font color="#ff2600" class="">-f</font> ~/short.conf </font></div>
<div class=""><font face="Menlo" class=""><br class="">
</font></div>
<div class=""><font face="Menlo" class="">rather than -c. It does not effect the error messages…. It was running on the full config not the stripped down one.</font></div>
<div class=""><font face="Menlo" class=""><br class="">
</font></div>
<div class=""><font face="Menlo" class="">R</font></div>
<div class=""><br class="">
<br class="">
<blockquote type="cite" class="">On 29/03/2020, at 4:03 PM, Russell Fulton <<a href="mailto:r.fulton@auckland.ac.nz" class="">r.fulton@auckland.ac.nz</a>> wrote:<br class="">
<br class="">
I have had this config running happily for several months and today I tried to add the geoip2 plugin and managed to break everything.<br class="">
<br class="">
I first realised that I needed to recompile syslog-ng to get the geoip2 plugin but forgot I was running a locally compiled version .<br class="">
<br class="">
I tried apt install syslog-ng-mod-geoip and this broke things badly and I have been trying to recover for the last few hours :(<br class="">
<br class="">
The issue is that any attempt to start syslog-ng with the original config that includes a destination of elasticsearch_http get an error:<br class="">
<br class="">
Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.22/etc/syslog-ng.conf:33:3-33:21<br class="">
<br class="">
Judging from what I found on th ’Net this is what one would expect if the syslog-ng version was prior to 3.21 and did not support the http dest.<br class="">
<br class="">
I first tried recompiling and installing 3.21 (in a different dir) with geoip2 enabled. That went with out errors but still would not process my config.<br class="">
<br class="">
Aside: At this point I found that in spite of having the geoip2 module included, syslog-ng failed to find it when I copied the setup from the manual. long story short the manual says the module is ‘geoip2’ when in fact it is ‘geoip2-plugin’ and that is the
name you have to use in the @module. Weird!<br class="">
<br class="">
I then compiled 3.22 since I had the tar file and this behaved the same.<br class="">
<br class="">
rful011@secmgrprd02:~$ cat short.conf <br class="">
@version: 3.21<br class="">
@include "scl.conf"<br class="">
<br class="">
@module geoip2-plugin<br class="">
<br class="">
source s_loghost {<br class="">
network( transport("tcp") flags(no-multi-line) port(1514) keep-alive(yes));<br class="">
};<br class="">
<br class="">
<br class="">
destination d_elastic {<br class="">
elasticsearch_http(<br class="">
index("auth_${YEAR}.${MONTH}.${DAY}")<br class="">
type("auth")<br class="">
persist-name("auth")<br class="">
template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")<br class="">
url("<a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsecesprd01.its.auckland.ac.nz%3A9200%2F_bulk&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C79eabe24c46e40c8e4d108d7d416a7ad%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637211066100185486&sdata=zq1UUYGy3IE2TF4Lb%2FEwiWWlbZDw5GrYbsuWB2Pe1e0%3D&reserved=0" originalsrc="http://secesprd01.its.auckland.ac.nz:9200/_bulk" shash="y8NYlhZz7N0H+r4lbpgiW0QRHU5zLSMWjFiaWyfBxfwFMfJEA0Q1TTAzJgu6zXPjLLmCkKQZQyqg7XZS5+0FxmOEOFqH2GfMXHnYk+0mltzS/0vqdwunEtxFXEU6do+HkBaZh+D9FzpYSJ/syBFM7bp3OjKYNmXrN4SWPGtW9G8=" class="">http://secesprd01.its.auckland.ac.nz:9200/_bulk</a>")<br class="">
);<br class="">
};<br class="">
<br class="">
log {<br class="">
source(s_loghost);<br class="">
destination(d_elastic );<br class="">
flags(flow-control);<br class="">
};<br class="">
<br class="">
rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng -s -c ~/short.conf <br class="">
Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.21/etc/syslog-ng.conf:33:3-33:21:<br class="">
28 file( "/home/rful011/test.log" template(t_db_parsed_test));<br class="">
29 };<br class="">
30 <br class="">
31 <br class="">
32 destination d_elastic {<br class="">
33----> elasticsearch_http(<br class="">
33----> ^^^^^^^^^^^^^^^^^^<br class="">
34 index("auth_${YEAR}.${MONTH}.${DAY}")<br class="">
35 type("auth")<br class="">
36 persist-name("auth")<br class="">
37 template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")<br class="">
38 url("http://secesprd01.its.auckland.ac.nz:9200/_bulk”)<br class="">
<br class="">
rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng --version<br class="">
syslog-ng 3 (3.21.1)<br class="">
Config version: 3.21<br class="">
Installer-Version: 3.21.1<br class="">
Revision: <br class="">
Compile-Date: Mar 29 2020 11:57:39<br class="">
Module-Directory: /usr/local/syslog-ng-3.21/lib/syslog-ng<br class="">
Module-Path: /usr/local/syslog-ng-3.21/lib/syslog-ng<br class="">
Include-Path: /usr/local/syslog-ng-3.21/share/syslog-ng/include<br class="">
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip2-plugin,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml<br class="">
Enable-Debug: off<br class="">
Enable-GProf: off<br class="">
Enable-Memtrace: off<br class="">
Enable-IPv6: on<br class="">
Enable-Spoof-Source: off<br class="">
Enable-TCP-Wrapper: off<br class="">
Enable-Linux-Caps: off<br class="">
Enable-Systemd: off<br class="">
<br class="">
<br class="">
<br class="">
______________________________________________________________________________<br class="">
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br class="">
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<br class="">
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<br class="">
<br class="">
</blockquote>
<br class="">
</div>
</div>
</div>
</div>
</div>
</body>
</html>