[syslog-ng] Error initializing message pipeline;

[Fabien Wernli]

Hi Damian,

On Fri, Jan 20, 2017 at 02:02:53PM +0000, Damian Bell wrote:

> “non-interesting” traffic at the syslog-ng level (if so, what is the
> best practice?) or do so at the Kibana level? In terms of transportation
> from syslog-ng into ES, does anyone have any tips or pointers as to
> the best way of formatting Cisco switch/firewall/router logs to best be
> utilised within ES/Kibana?

I'd say it depends on your ability to store the full monty.
If you can afford it, then you'll be happy to use the REST API to filter out relevant results.
If you can't, you'll have to filter out stuff upstream using syslog-ng, and
use ES to query what's left.

That being said, in any case you'll want the data to be in a nice structured
format, so that you can easily and efficiently filter it using either
syslog-ng or kibana. And for that to work, you'll have to parse the lot,
because you can't rely on regexp or lucene full-text searches for everything
(unless you're extremely rich and have exabytes of SSDs lying around).

So my suggested plan would be:

1. write parsers in syslog-ng to structure your logs into name/value pairs
   - patterndb
   - csv
   - kv-parser
   - python
   - …
2. use kibana AND/OR syslog_ng to filter using the key/value pairs you created
3. goto 1.

We started out using patterndb and are extremely happy with it.
But today you have a large choice of available parsers to extract keys with.
Today we use a combination of many, and to choose one for a given use-case
we usually try with patterndb, csv or kv-parser, and for anything more
complex we use python.

I'd be happy to share any particular use-case with you.

Cheers