[syslog-ng] patterndb action rate limiting

[Evan Rempel]

The statistics may address my current needs, however, I think there will be many more
cases where multiple actions may benefit from rate limiting on different scopes.

For instance, if I wanted to feed a program() destination that sends e-mail,
I may wish to rate limit for each destination address. This can't be done inside patterndb
at all since you can not select an arbitrary string as the context_scope, nor could you
send email to two different destinations for the same match and have them rate limited
differently.

A real world example of this is that we are going to tag messages with "Network Operations Center"
tags, so they can then be sent to alerting systems and ticketing systems. We will
want to rate limit these events on a "destination address/problem keywords (ticket queue/host/app/state)"
type of context_scope. We will have to do this external to syslog-ng for now, but the problem with
that is if syslog-ng feeds data to the external program without a rate limit, the program may
not be able to keep up, and some events will get dropped.

If an arbitrary string, possibly with macro/variable expansion, the rate limiting feature
would be much more versatile.

Evan.

Fekete Róbert wrote:
>  
> On Monday, November 28, 2011 21:11 CET, Evan Rempel <erempel at uvic.ca> wrote: 
>  
>> Yes, you get the idea. Multiple actions for a single matched message with different
>> rate limits on each.
>>
> That should work.
> 
>> Sadly, I have not yet gotten to the statistics portion of this work, so I can't answer your question,
>> and probably will not be able to for another month (I'm going somewhere warm. Happy me:-)
> 
> Thinking about it, patterndb-related statistics are available in our commercial syslog-ng server appliance (syslog-ng Store Box), so I believe that these stats are available in OSE as well on stats_level 3.
> 
> Happy Holidays!
> 
> Robert
> 
>> Evan.
>>
>> Fekete Róbert wrote:
>>>  
>>> On Monday, November 28, 2011 19:44 CET, Evan Rempel <erempel at uvic.ca> wrote: 
>>>  
>>>> The patterndb has a great feature for limiting the rate that actions are executed.
>>>> The rate limit needs to have a scope within which to count the action events that will
>>>> in turn determine when the limit has been reached and throttling is to occur.
>>>>
>>>> The problem with the current patterndb format is that the context is defined in
>>>> the rule, which means that I can NOT have different rate contexts for multiple
>>>> actions for the same rule :-(
>>>>
>>>> Am I misreading the patterndb specification?
>>> Hi, 
>>>
>>> If I got that right, you would like to have multiple actions for a rule, and define different rate limits for each action. Like:
>>> <actions>
>>>   <action1 ratelimit1>
>>>   <action2 ratelimit2>
>>> </actions>
>>>
>>> This should be possible in OSE 3.3, because the rate limit is an option of the action object, not of the actions group.
>>> See https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/index.html-single.html#patterndb-schema-actions
>>>
>>>> Another not is that I have a case where I want to rate limit an action based
>>>> on the rule ID. In my case I don't care about which host or process, or process ID
>>>> the message came from.
>>>>
>>>> I know that Balazs will ask for a use case to determine if I am "insane" or
>>>> if this context-scope of MESSAGE will be of general use.
>>>>
>>>> What I am trying to do is determine which patterns are being matched.
>>>> I would like to generate an action for every match and feed that to a program
>>>> that will record the date/time that the matching pattern last matched a message.
>>>>
>>>> If the pattern has not seen a match for a year, then I can (and probably should)
>>>> remove it from my pattern database.
>>>>
>>>> I don't want my program to have to deal with a rule_id for *every* match.
>>>> I could safely throttle the matched information to 1 every 10 minutes, or perhaps every
>>>> hour. That would still let me know if a pattern is "active" and should be kept in
>>>> the pattern database.
>>>>
>>>> Does this sound reasonable (or am I really insane :-) ?
>>> That's right, this is not currently possible, and using suppress on the triggered messages would not work, because with many rules the triggered messages would be intermixed. And unfortunately correlation does not work either (like adding identical messages to a $RULE_ID context), because every message would update the context timeout, and the contexts of high-traffic patterns would be kept indefinitely (or at least until they eat up the memory). With that said, I see a point for this use case :)
>>>
>>> One more thought on this: could you try to increase the stats_level() global option to 3, and see if the statistics contain details for patterndb matches? It might be possible that they do (statistics is sadly underdocumented in the admin guide - my fault, sorry). If this works, then you'd only have to query the statistics every ten minutes or so, and process the needed parts with a script.
>>>
>>> Robert
>>>
>>>> Evan.
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>  
>>>  
>>>  
>>>  
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>
>> -- 
>> Evan Rempel                               erempel at uvic.ca
>> Senior Systems Administrator                 250.721.7691
>> Unix Services, University Systems, University of Victoria
>>
>  
>  
>  
>  
> 
> 


-- 
Evan Rempel                               erempel at uvic.ca
Senior Systems Administrator                 250.721.7691
Unix Services, University Systems, University of Victoria