You could also try LogZilla - it automatically deduplicates messages. ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Thu, May 22, 2014 at 4:54 PM, Evade Flow <evadeflow@gmail.com> wrote:
Thanks for the response, it really helped me see how syslog-ng can break apart an input stream and re-assemble it. Seems very powerful!
With regard to filtering/pattern matching: do I need to know the patterns to be matched in advance? In my case, unfortunately, I do not. Some of these apps were written by people I've never met, and the SLOC count for all apps taken together is well over 1 million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those?
On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs:
Here's the way I'd try doing it, based on your example:
1) Use filters to separate the "myapp: Battery voltage is N volts" message flow from all other messages. 2) Tag the "Regular flow" of messages using e.g. the tag "keep" Optionally tag the other stream using "drop" 3) Parse the "myapp" messages using a patterndb parser, correlating all messages from the same host, then trigger an action upon timeout or reaching threshold (use context-length macro). The generated message can contain useful information from the context, e.g. number of similar messages (CONTEXT_LENGTH), average of voltages (would probably need some template function acrobatics), etc. 4) Tag the action generated message using "keep" 5) Filter out all messages not having the tag "keep". This way the individual "myapp" messages won't be logged. Only the "regular flow" of messages and the triggered messages will.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq