[syslog-ng] Suppress almost-identical messages?

[Orangepeel Beef]

you may want to take a look at this presentation
http://www.occam.com/sa/CentralizedLogging2012.pdf   SEC deduping does not
require you to know the patterns ahead of time.  But YMMV ;)


On Thu, May 22, 2014 at 3:38 PM, Clayton Dukes <cdukes at gmail.com> wrote:

> You could also try LogZilla - it automatically deduplicates messages.
>
>
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________
>
>
> On Thu, May 22, 2014 at 4:54 PM, Evade Flow <evadeflow at gmail.com> wrote:
>
>> Thanks for the response, it really helped me see how syslog-ng can break
>> apart an input stream and re-assemble it. Seems very powerful!
>>
>> With regard to filtering/pattern matching: do I need to know the patterns
>> to be matched in advance? In my case, unfortunately, I do not. Some of
>> these apps were written by people I've never met, and the SLOC count for
>> all apps taken together is well over 1 million lines of code. I suppose I
>> could run the app hundreds of times under various loads, collect
>> representative log files, and build the pattern database from those?
>>
>>
>>
>> On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli at in2p3.fr> wrote:
>>
>>> Hi,
>>>
>>> On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
>>> > Does syslog-ng support suppression of almost-but-not-quite identical
>>> > messages? It would be nice to see something like this in the logs:
>>>
>>> Here's the way I'd try doing it, based on your example:
>>>
>>> 1) Use filters to separate the "myapp: Battery voltage is N volts"
>>> message
>>>    flow from all other messages.
>>> 2) Tag the "Regular flow" of messages using e.g. the tag "keep"
>>>    Optionally tag the other stream using "drop"
>>> 3) Parse the "myapp" messages using a patterndb parser, correlating all
>>>    messages from the same host, then trigger an action upon timeout or
>>>    reaching threshold (use context-length macro). The generated message
>>> can
>>>    contain useful information from the context, e.g. number of similar
>>>    messages (CONTEXT_LENGTH), average of voltages (would probably need
>>> some
>>>    template function acrobatics), etc.
>>> 4) Tag the action generated message using "keep"
>>> 5) Filter out all messages not having the tag "keep". This way the
>>>    individual "myapp" messages won't be logged. Only the "regular flow"
>>> of
>>>    messages and the triggered messages will.
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140522/b334a035/attachment.htm