You could also try LogZilla - it automatically deduplicates messages.

______________________________________________________________

Clayton Dukes
______________________________________________________________

On Thu, May 22, 2014 at 4:54 PM, Evade Flow <evadeflow@gmail.com> wrote:

Thanks for the response, it really helped me see how syslog-ng can break apart an input stream and re-assemble it. Seems very powerful!

With regard to filtering/pattern matching: do I need to know the patterns to be matched in advance? In my case, unfortunately, I do not. Some of these apps were written by people I've never met, and the SLOC count for all apps taken together is well over 1 million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those?

On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli@in2p3.fr> wrote:

Hi,

On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
> Does syslog-ng support suppression of almost-but-not-quite identical
> messages? It would be nice to see something like this in the logs:

Here's the way I'd try doing it, based on your example:

1) Use filters to separate the "myapp: Battery voltage is N volts" message
   flow from all other messages.
2) Tag the "Regular flow" of messages using e.g. the tag "keep"
   Optionally tag the other stream using "drop"
3) Parse the "myapp" messages using a patterndb parser, correlating all
   messages from the same host, then trigger an action upon timeout or
   reaching threshold (use context-length macro). The generated message can
   contain useful information from the context, e.g. number of similar
   messages (CONTEXT_LENGTH), average of voltages (would probably need some
   template function acrobatics), etc.
4) Tag the action generated message using "keep"
5) Filter out all messages not having the tag "keep". This way the
   individual "myapp" messages won't be logged. Only the "regular flow" of
   messages and the triggered messages will.

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq