Worked out http mode only supported not long after receiving the mail from Fabien, and now all working (thanks!).

A more “generalised” question – I’ve used syslog-ng for years as a network engineer to receive Cisco network device input and output it to file, both as individual host data and also a collected “all” file on which I’ve used a very simple “swatch” implementation to both screen out noise, and also highlight interesting network events (routing convergence etc), as per config below. I’d like to replicate this somewhat with the syslog-ng/ES/Kibana build I now have, but I’m wondering the best way of doing it – should I filter “non-interesting” traffic at the syslog-ng level (if so, what is the best practice?) or do so at the Kibana level? In terms of transportation from syslog-ng into ES, does anyone have any tips or pointers as to the best way of formatting Cisco switch/firewall/router logs to best be utilised within ES/Kibana?

Thank you very much in advance.

From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Laszlo Budai
Sent: 18 January 2017 05:40
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>; Fabien Wernli <wernli@in2p3.fr>
Subject: Re: [syslog-ng] Error initializing message pipeline;

hi,

we support ES5.x only via http mode.

regards,

Laszlo Budai

_____________________________
From: Scot <scotrn@gmail.com>
Sent: Wednesday, January 18, 2017 3:33 AM
Subject: Re: [syslog-ng] Error initializing message pipeline;
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>, Fabien Wernli <wernli@in2p3.fr>

Is client-mode("transport") now supported with ES 5.1? I thought it was only http mode for ES 5.

I got pipeline error then switched to http thinking it was the transport mode. http worked fine.

On Tue, Jan 17, 2017 at 9:58 AM, Fabien Wernli <wernli@in2p3.fr> wrote:

Hi Damian,

You need to specify the location to your elasticsearch installation, i.e.
where the .jar files are installed.
If you're using the official packages from elastic.co, they are most likely
located here: /usr/share/elasticsearch/lib/

So your config ought to look like the following instead:

source s_syslog { udp(ip(0.0.0.0) port(514)); };

destination d_elastic {
elasticsearch2(
client-lib-dir("/usr/share/elasticsearch/lib/")
index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
type("test")
cluster("someserver")
client-mode("transport")
template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
time-zone("UTC")
);
};

Moreover, you might want to set the destination's timezone to UTC too, or
you'll have surprises in kibana around midnight UTC: time-zone("UTC")

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq