<div dir="ltr">Thanks for mentioning SEC, and the link to the (quite good) presentation. We've decided that pre-processing the logs is too much work, and probably a bad idea, so we're just going to attempt to ensure that none of the apps are too chatty, and rely on our log rotation scheme to deal with the fallout from any corner cases that cause the logs to 'blow up'. But SEC looks like a slam dunk for post-processing/analyzing logs collected from devices in the field.<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 22, 2014 at 7:00 PM, Orangepeel Beef <span dir="ltr"><<a href="mailto:orangepeelbeef@gmail.com" target="_blank">orangepeelbeef@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">you may want to take a look at this presentation <a href="http://www.occam.com/sa/CentralizedLogging2012.pdf" target="_blank">http://www.occam.com/sa/CentralizedLogging2012.pdf</a> SEC deduping does not require you to know the patterns ahead of time. But YMMV ;)<br>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 22, 2014 at 3:38 PM, Clayton Dukes <span dir="ltr"><<a href="mailto:cdukes@gmail.com" target="_blank">cdukes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">You could also try LogZilla - it automatically deduplicates messages.<span><font color="#888888"><div>
<br></div></font></span></div><div class="gmail_extra"><span><font color="#888888"><br clear="all"><div>______________________________________________________________ <br><br>
Clayton Dukes<br>______________________________________________________________</div></font></span><div><div>
<br><br><div class="gmail_quote">On Thu, May 22, 2014 at 4:54 PM, Evade Flow <span dir="ltr"><<a href="mailto:evadeflow@gmail.com" target="_blank">evadeflow@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div>Thanks for the response, it really helped me see how syslog-ng
can break apart an input stream and re-assemble it. Seems very
powerful!<br><br></div>With regard to filtering/pattern matching: do I
need to know the patterns to be matched in advance? In my case,
unfortunately, I do not. Some of these apps were written by people I've
never met, and the SLOC count for all apps taken together is well over 1
million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those?<div><div><br><br><br><div class="gmail_quote">
On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<div><br>
On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:<br>
> Does syslog-ng support suppression of almost-but-not-quite identical<br>
> messages? It would be nice to see something like this in the logs:<br>
<br>
</div>Here's the way I'd try doing it, based on your example:<br>
<br>
1) Use filters to separate the "myapp: Battery voltage is N volts" message<br>
flow from all other messages.<br>
2) Tag the "Regular flow" of messages using e.g. the tag "keep"<br>
Optionally tag the other stream using "drop"<br>
3) Parse the "myapp" messages using a patterndb parser, correlating all<br>
messages from the same host, then trigger an action upon timeout or<br>
reaching threshold (use context-length macro). The generated message can<br>
contain useful information from the context, e.g. number of similar<br>
messages (CONTEXT_LENGTH), average of voltages (would probably need some<br>
template function acrobatics), etc.<br>
4) Tag the action generated message using "keep"<br>
5) Filter out all messages not having the tag "keep". This way the<br>
individual "myapp" messages won't be logged. Only the "regular flow" of<br>
messages and the triggered messages will.<br>
<div><div><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br></div></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
</div></div><br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>