<div dir="ltr">you may want to take a look at this presentation <a href="http://www.occam.com/sa/CentralizedLogging2012.pdf">http://www.occam.com/sa/CentralizedLogging2012.pdf</a> SEC deduping does not require you to know the patterns ahead of time. But YMMV ;)<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 22, 2014 at 3:38 PM, Clayton Dukes <span dir="ltr"><<a href="mailto:cdukes@gmail.com" target="_blank">cdukes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">You could also try LogZilla - it automatically deduplicates messages.<span class="HOEnZb"><font color="#888888"><div>
<br></div></font></span></div><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><br clear="all"><div>______________________________________________________________ <br><br>
Clayton Dukes<br>______________________________________________________________</div></font></span><div><div class="h5">
<br><br><div class="gmail_quote">On Thu, May 22, 2014 at 4:54 PM, Evade Flow <span dir="ltr"><<a href="mailto:evadeflow@gmail.com" target="_blank">evadeflow@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div>Thanks for the response, it really helped me see how syslog-ng
can break apart an input stream and re-assemble it. Seems very
powerful!<br><br></div>With regard to filtering/pattern matching: do I
need to know the patterns to be matched in advance? In my case,
unfortunately, I do not. Some of these apps were written by people I've
never met, and the SLOC count for all apps taken together is well over 1
million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those?<div><div><br><br><br><div class="gmail_quote">
On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<div><br>
On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:<br>
> Does syslog-ng support suppression of almost-but-not-quite identical<br>
> messages? It would be nice to see something like this in the logs:<br>
<br>
</div>Here's the way I'd try doing it, based on your example:<br>
<br>
1) Use filters to separate the "myapp: Battery voltage is N volts" message<br>
flow from all other messages.<br>
2) Tag the "Regular flow" of messages using e.g. the tag "keep"<br>
Optionally tag the other stream using "drop"<br>
3) Parse the "myapp" messages using a patterndb parser, correlating all<br>
messages from the same host, then trigger an action upon timeout or<br>
reaching threshold (use context-length macro). The generated message can<br>
contain useful information from the context, e.g. number of similar<br>
messages (CONTEXT_LENGTH), average of voltages (would probably need some<br>
template function acrobatics), etc.<br>
4) Tag the action generated message using "keep"<br>
5) Filter out all messages not having the tag "keep". This way the<br>
individual "myapp" messages won't be logged. Only the "regular flow" of<br>
messages and the triggered messages will.<br>
<div><div><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br></div></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>