[syslog-ng] Cannot send Syslog-ng to Elasticsearch

[Allen Olivas]

Ok so my attempt to build and add the certificates and CA still did not work. On whim I pointed the TLS statement to the existing demo certs from searchguard. 

After restarting syslog-ng I found the service was still running (I don't know why it worked this time and not the million other times I tried it) but data is still not traversing to elasticsearch due to (I believe) two new errors. These two errors are most likely related and not separate errors altogether. 

Here are the two errors I'm seeing: 
1: From /var/log/message - Server returned with a 4XX (client errors) status code, which means we are not authorized or the URL is not found.;
2: From /var/log/error - syslog-ng[18498]: Message(s) dropped while sending message to destination; driver='d_elastic#0', worker_index='1', time_reopen='60', batch_size='3'

I also see the following from 'syslog-ng -c syslog-ng.service -e -v'

Unknown argument, adding it to __VARARGS__; argument='tls', value='\x0a      ca-file("/etc/elasticsearch/root-ca.pem")\x0a      cert-file("/etc/elasticsearch/esnode.pem")\x0a      key-file("/etc/elasticsearch/esnode-key.pem")\x0a      peer-verify(yes)\x0a      ', reference='/etc/syslog-ng/syslog-ng.conf:83:3'

I don't know why that would be an unknown argument but maybe that's the problem  right there? 

Thoughts? Thanks for all your support everyone!

-----Original Message-----
From: Fabien Wernli <wernli at in2p3.fr> 
Sent: Thursday, July 11, 2019 2:00 AM
To: Allen Olivas <allen.olivas at infodefense.com>
Cc: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: RE: RE: RE: [syslog-ng] Cannot send Syslog-ng to Elasticsearch

Hi,

On Wed, Jul 10, 2019 at 08:22:38PM +0000, Allen Olivas wrote:
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
> tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      1032/systemd-resolv
> tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1874/sshd
> tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      2145/master
> tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      13557/sshd: aolivas
> tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      15586/sshd: aolivas
> tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      1314/node
> tcp        0      0 0.0.0.0:1515            0.0.0.0:*               LISTEN      2329/ossec-authd
> tcp6       0      0 :::9200                 :::*                    LISTEN      1738/java
> tcp6       0      0 :::9300                 :::*                    LISTEN      1738/java
> tcp6       0      0 :::22                   :::*                    LISTEN      1874/sshd
> tcp6       0      0 :::55000                :::*                    LISTEN      1734/nodejs
> tcp6       0      0 :::25                   :::*                    LISTEN      2145/master
> tcp6       0      0 ::1:6010                :::*                    LISTEN      13557/sshd: aolivas
> tcp6       0      0 ::1:6011                :::*                    LISTEN      15586/sshd: aolivas

It seems to me your ES is listening on ipv6 only.
Please retry after setting the following in your elasticsearch.yml:

  network.host:
  - 127.0.0.1

And then curl to 127.0.0.1 explicitly (localhost may resolve to ::1)