[syslog-ng] Error initializing message pipeline;

[Damian Bell]

Worked out http mode only supported not long after receiving the mail from Fabien, and now all working (thanks!).

A more “generalised” question – I’ve used syslog-ng for years as a network engineer to receive Cisco network device input and output it to file, both as individual host data and also a collected “all” file on which I’ve used a very simple “swatch” implementation to both screen out noise, and also highlight interesting network events (routing convergence etc), as per config below. I’d like to replicate this somewhat with the syslog-ng/ES/Kibana build I now have, but I’m wondering the best way of doing it – should I filter “non-interesting” traffic at the syslog-ng level (if so, what is the best practice?) or do so at the Kibana level? In terms of transportation from syslog-ng into ES, does anyone have any tips or pointers as to the best way of formatting Cisco switch/firewall/router logs to best be utilised within ES/Kibana?

Thank you very much in advance.




Damian Bell
Infrastructure Engineer | Support | H Clarkson & Co Ltd

Email: Damian.Bell at clarksons.com<mailto:Damian.Bell at clarksons.com>


From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Laszlo Budai
Sent: 18 January 2017 05:40
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>; Fabien Wernli <wernli at in2p3.fr>
Subject: Re: [syslog-ng] Error initializing message pipeline;

hi,

we support ES5.x only via http mode.

regards,
Laszlo Budai
_____________________________
From: Scot <scotrn at gmail.com<mailto:scotrn at gmail.com>>
Sent: Wednesday, January 18, 2017 3:33 AM
Subject: Re: [syslog-ng] Error initializing message pipeline;
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>, Fabien Wernli <wernli at in2p3.fr<mailto:wernli at in2p3.fr>>



Is client-mode("transport") now supported with ES 5.1?   I thought it was only http mode for ES 5.

I got pipeline error then switched to http thinking it was the transport mode. http worked fine.


On Tue, Jan 17, 2017 at 9:58 AM, Fabien Wernli <wernli at in2p3.fr<mailto:wernli at in2p3.fr>> wrote:
Hi Damian,

You need to specify the location to your elasticsearch installation, i.e.
where the .jar files are installed.
If you're using the official packages from elastic.co<http://elastic.co>, they are most likely
located here: /usr/share/elasticsearch/lib/

So your config ought to look like the following instead:

    source s_syslog { udp(ip(0.0.0.0) port(514)); };

    destination d_elastic {
      elasticsearch2(
        client-lib-dir("/usr/share/elasticsearch/lib/")
        index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
        type("test")
        cluster("someserver")
        client-mode("transport")
        template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
        time-zone("UTC")
      );
    };

Moreover, you might want to set the destination's timezone to UTC too, or
you'll have surprises in kibana around midnight UTC: time-zone("UTC")

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



________________________________
This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence.

Emails may be monitored.

Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/>
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170120/5355ac43/attachment.html>