[syslog-ng] Suppress almost-identical messages?
evadeflow at gmail.com
Thu May 22 22:54:05 CEST 2014
Thanks for the response, it really helped me see how syslog-ng can break
apart an input stream and re-assemble it. Seems very powerful!
With regard to filtering/pattern matching: do I need to know the patterns
to be matched in advance? In my case, unfortunately, I do not. Some of
these apps were written by people I've never met, and the SLOC count for
all apps taken together is well over 1 million lines of code. I suppose I
could run the app hundreds of times under various loads, collect
representative log files, and build the pattern database from those?
On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli at in2p3.fr> wrote:
> On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
> > Does syslog-ng support suppression of almost-but-not-quite identical
> > messages? It would be nice to see something like this in the logs:
> Here's the way I'd try doing it, based on your example:
> 1) Use filters to separate the "myapp: Battery voltage is N volts" message
> flow from all other messages.
> 2) Tag the "Regular flow" of messages using e.g. the tag "keep"
> Optionally tag the other stream using "drop"
> 3) Parse the "myapp" messages using a patterndb parser, correlating all
> messages from the same host, then trigger an action upon timeout or
> reaching threshold (use context-length macro). The generated message can
> contain useful information from the context, e.g. number of similar
> messages (CONTEXT_LENGTH), average of voltages (would probably need some
> template function acrobatics), etc.
> 4) Tag the action generated message using "keep"
> 5) Filter out all messages not having the tag "keep". This way the
> individual "myapp" messages won't be logged. Only the "regular flow" of
> messages and the triggered messages will.
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the syslog-ng