[syslog-ng] Error initializing message pipeline;

[Evan Rempel]

Fabian, I would be interested in seeing your patterndb file for the 
cisco logs. Our problem is that the cisco devices don't really log a 
program name, which makes using patterndb quite difficult.

Can you share in or out of band?

Evan

On 01/20/2017 07:04 AM, Fabien Wernli wrote:
> Hi Damian,
>
> On Fri, Jan 20, 2017 at 02:02:53PM +0000, Damian Bell wrote:
>
>> “non-interesting” traffic at the syslog-ng level (if so, what is the
>> best practice?) or do so at the Kibana level? In terms of transportation
>> from syslog-ng into ES, does anyone have any tips or pointers as to
>> the best way of formatting Cisco switch/firewall/router logs to best be
>> utilised within ES/Kibana?
> I'd say it depends on your ability to store the full monty.
> If you can afford it, then you'll be happy to use the REST API to filter out relevant results.
> If you can't, you'll have to filter out stuff upstream using syslog-ng, and
> use ES to query what's left.
>
> That being said, in any case you'll want the data to be in a nice structured
> format, so that you can easily and efficiently filter it using either
> syslog-ng or kibana. And for that to work, you'll have to parse the lot,
> because you can't rely on regexp or lucene full-text searches for everything
> (unless you're extremely rich and have exabytes of SSDs lying around).
>
> So my suggested plan would be:
>
> 1. write parsers in syslog-ng to structure your logs into name/value pairs
>     - patterndb
>     - csv
>     - kv-parser
>     - python
>     - …
> 2. use kibana AND/OR syslog_ng to filter using the key/value pairs you created
> 3. goto 1.
>
> We started out using patterndb and are extremely happy with it.
> But today you have a large choice of available parsers to extract keys with.
> Today we use a combination of many, and to choose one for a given use-case
> we usually try with patterndb, csv or kv-parser, and for anything more
> complex we use python.
>
> I'd be happy to share any particular use-case with you.
>
> Cheers