[syslog-ng] Cannot send Syslog-ng to Elasticsearch

[Allen Olivas]

Hello,

I'm trying to authenticate with searchguard. The destination I have specified includes the tls() configuration you suggested:
destination d_elastic {
  elasticsearch-http(
    url("https://127.0.0.1:9200/_bulk")
    index("logs-${YEAR}.${MONTH}.${DAY}")
    type("syslog")
    client-mode ("transport")
    tls(
      ca-file("/path/to/ca.pem")
      cert-file("/path/to/syslog_ng.crt.pem")
      key-file("/path/to/syslog_ng.key.pem")
      peer-verify(yes)
      )
   );
};

My problem now is it  still doesn't seem to authenticate or work with elasticsearch. 

Should I have an entry in the elasticsearch.yml? Searchguard has already been configured for elasticsearch and kibana. Also is your elastic-http-plugin.conf referencing the yml file or the client-mode ("searchguard")? I'm not entirely sure what all needs to be configured. 

The specific errors I'm seeing are below:
[2019-07-10T01:44:39.077952] Server disconnected while preparing messages for sending, trying again; driver='d_elastic#0', location='#buffer:4:3', worker_index='0', time_reopen='60', batch_size='3'
[2019-07-10T01:44:39.100211] curl: error sending HTTP request; url='https://127.0.0.1:9200/_bulk', error='Problem with the local SSL certificate', worker_index='3', driver='d_elastic#0', location='#buffer:4:3'
[2019-07-10T01:44:39.100230] Target server down, but no alternative server available. Falling back to retrying after time-reopen(); url='https://127.0.0.1:9200/_bulk', worker_index='3', driver='d_elastic#0', location='#buffer:4:3'-----

When I check the indices available I do not see anything created for syslog-ng. I feel like its almost configured so I'm pretty excited to get this completed and documented on my end. 

Thanks again for all the support.

Original Message-----
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Fabien Wernli
Sent: Wednesday, July 10, 2019 1:20 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Cannot send Syslog-ng to Elasticsearch

Hi,

On Tue, Jul 09, 2019 at 09:56:50PM +0000, Allen Olivas wrote:
> Ok I've got it configured but now its I think its not building the index and updating elasticsearch because of HTTPS and authentication. I have searchguard set up for elasticsearch and kibana. I'm assuming I need Syslog-ng to use the SSL certs searchguard has in place for elasticsearch. 

You can use almost any authentication method supported by Searchguard.
We use client certificates for syslog-ng, and here's how the config looks
like:

  destination d_coloss {
    elasticsearch-http(
       url("https://node01:9200/_bulk" "https://node02:9200/_bulk")
       index("syslog-${YEAR}-${MONTH}-${DAY}")
       time-zone("UTC")
       type("")
       workers(4)
       batch_lines(128)
       batch_timeout(10000)
       timeout(100)
       tls(
         ca-file("/path/to/ca.pem")
         cert-file("/path/to/syslog_ng.crt.pem")
         key-file("/path/to/syslog_ng.key.pem")
         peer-verify(yes)
       )
    );
  };

And here are the searchguard permissions for the syslog-ng user's role:

  sg_role_syslog_ng:
    indices:
      "syslog":
        "*":
        - WRITE
        - CREATE_INDEX
        - indices:admin/mapping/put
    cluster:
    - CLUSTER_COMPOSITE_OPS
    - cluster:monitor/nodes/info
    - cluster:monitor/nodes/liveness
    - cluster:monitor/state

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq