[syslog-ng] Suppress almost-identical messages?

[Evade Flow]

Thanks for mentioning SEC, and the link to the (quite good) presentation.
We've decided that pre-processing the logs is too much work, and probably a
bad idea, so we're just going to attempt to ensure that none of the apps
are too chatty, and rely on our log rotation scheme to deal with the
fallout from any corner cases that cause the logs to 'blow up'.  But SEC
looks like a slam dunk for post-processing/analyzing logs collected from
devices in the field.


On Thu, May 22, 2014 at 7:00 PM, Orangepeel Beef
<orangepeelbeef at gmail.com>wrote:

> you may want to take a look at this presentation
> http://www.occam.com/sa/CentralizedLogging2012.pdf   SEC deduping does
> not require you to know the patterns ahead of time.  But YMMV ;)
>
>
> On Thu, May 22, 2014 at 3:38 PM, Clayton Dukes <cdukes at gmail.com> wrote:
>
>> You could also try LogZilla - it automatically deduplicates messages.
>>
>>
>> ______________________________________________________________
>>
>> Clayton Dukes
>> ______________________________________________________________
>>
>>
>> On Thu, May 22, 2014 at 4:54 PM, Evade Flow <evadeflow at gmail.com> wrote:
>>
>>> Thanks for the response, it really helped me see how syslog-ng can break
>>> apart an input stream and re-assemble it. Seems very powerful!
>>>
>>> With regard to filtering/pattern matching: do I need to know the
>>> patterns to be matched in advance? In my case, unfortunately, I do not.
>>> Some of these apps were written by people I've never met, and the SLOC
>>> count for all apps taken together is well over 1 million lines of code. I
>>> suppose I could run the app hundreds of times under various loads, collect
>>> representative log files, and build the pattern database from those?
>>>
>>>
>>>
>>> On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli at in2p3.fr> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
>>>> > Does syslog-ng support suppression of almost-but-not-quite identical
>>>> > messages? It would be nice to see something like this in the logs:
>>>>
>>>> Here's the way I'd try doing it, based on your example:
>>>>
>>>> 1) Use filters to separate the "myapp: Battery voltage is N volts"
>>>> message
>>>>    flow from all other messages.
>>>> 2) Tag the "Regular flow" of messages using e.g. the tag "keep"
>>>>    Optionally tag the other stream using "drop"
>>>> 3) Parse the "myapp" messages using a patterndb parser, correlating all
>>>>    messages from the same host, then trigger an action upon timeout or
>>>>    reaching threshold (use context-length macro). The generated message
>>>> can
>>>>    contain useful information from the context, e.g. number of similar
>>>>    messages (CONTEXT_LENGTH), average of voltages (would probably need
>>>> some
>>>>    template function acrobatics), etc.
>>>> 4) Tag the action generated message using "keep"
>>>> 5) Filter out all messages not having the tag "keep". This way the
>>>>    individual "myapp" messages won't be logged. Only the "regular flow"
>>>> of
>>>>    messages and the triggered messages will.
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140523/de170937/attachment.htm