[syslog-ng] db-parser reuse for multiple logs?

[Evan Rempel]

I know you are being facetious, but...

We generate our patterns programmatically from our inventory/asset tracking database.
We use the syslog-ng patterndb to apply metadata about our hosts and applications.

The metadata consists of:

1. Which unit in our organization is responsible for an host/application pair, so in the event that a log message should generate an alert/incident, our alerting layer knows which group to direct the alert to.

2. The role that the host plays (development, pre-production, production, BCP or Disaster recovery). That information is used to decide if alerts should go to pagers/cell phones, if they should go to e-mail or ticketing systems or if they should be ignored completely. Essentially a severity based on the role of the host.


What this all means is that we use syslog-ng patterdb against a template of "$HOST:$PROGRAM" and have patterns for all hosts/program combinations (not all combinations since there is a catchall pattern) that we have in our asset tracking system.

It is this large combination that makes the 20,000 pattern database.

We have the other 4,300 patterns that match the actual log lines and define what kind of event each log line is. We are currently matching > 98% of our log volume and classifying the messages as one of safe, alert, timer, heartbeat, collection, rate threshold or command line tool execution.

Just thought I would explain a little about how useful the patterndb is at our site.

Evan.

On 04/14/2015 02:08 AM, Fabien Wernli wrote:
> On Mon, Apr 13, 2015 at 08:37:56AM -0700, Evan Rempel wrote:
>> We process approximately 5,000 msg/sec and use two patterndb parsers. One with 20,000 patterns and another with 4.300 patterns. All messages go through both parsers.
> 25k patterns…
> …you're lying!
>