The statistics may address my current needs, however, I think there will be many more cases where multiple actions may benefit from rate limiting on different scopes. For instance, if I wanted to feed a program() destination that sends e-mail, I may wish to rate limit for each destination address. This can't be done inside patterndb at all since you can not select an arbitrary string as the context_scope, nor could you send email to two different destinations for the same match and have them rate limited differently. A real world example of this is that we are going to tag messages with "Network Operations Center" tags, so they can then be sent to alerting systems and ticketing systems. We will want to rate limit these events on a "destination address/problem keywords (ticket queue/host/app/state)" type of context_scope. We will have to do this external to syslog-ng for now, but the problem with that is if syslog-ng feeds data to the external program without a rate limit, the program may not be able to keep up, and some events will get dropped. If an arbitrary string, possibly with macro/variable expansion, the rate limiting feature would be much more versatile. Evan. Fekete Róbert wrote:
On Monday, November 28, 2011 21:11 CET, Evan Rempel <erempel@uvic.ca> wrote:
Yes, you get the idea. Multiple actions for a single matched message with different rate limits on each.
That should work.
Sadly, I have not yet gotten to the statistics portion of this work, so I can't answer your question, and probably will not be able to for another month (I'm going somewhere warm. Happy me:-)
Thinking about it, patterndb-related statistics are available in our commercial syslog-ng server appliance (syslog-ng Store Box), so I believe that these stats are available in OSE as well on stats_level 3.
Happy Holidays!
Robert
Evan.
Fekete Róbert wrote:
On Monday, November 28, 2011 19:44 CET, Evan Rempel <erempel@uvic.ca> wrote:
The patterndb has a great feature for limiting the rate that actions are executed. The rate limit needs to have a scope within which to count the action events that will in turn determine when the limit has been reached and throttling is to occur.
The problem with the current patterndb format is that the context is defined in the rule, which means that I can NOT have different rate contexts for multiple actions for the same rule :-(
Am I misreading the patterndb specification? Hi,
If I got that right, you would like to have multiple actions for a rule, and define different rate limits for each action. Like: <actions> <action1 ratelimit1> <action2 ratelimit2> </actions>
This should be possible in OSE 3.3, because the rate limit is an option of the action object, not of the actions group. See https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid...
Another not is that I have a case where I want to rate limit an action based on the rule ID. In my case I don't care about which host or process, or process ID the message came from.
I know that Balazs will ask for a use case to determine if I am "insane" or if this context-scope of MESSAGE will be of general use.
What I am trying to do is determine which patterns are being matched. I would like to generate an action for every match and feed that to a program that will record the date/time that the matching pattern last matched a message.
If the pattern has not seen a match for a year, then I can (and probably should) remove it from my pattern database.
I don't want my program to have to deal with a rule_id for *every* match. I could safely throttle the matched information to 1 every 10 minutes, or perhaps every hour. That would still let me know if a pattern is "active" and should be kept in the pattern database.
Does this sound reasonable (or am I really insane :-) ? That's right, this is not currently possible, and using suppress on the triggered messages would not work, because with many rules the triggered messages would be intermixed. And unfortunately correlation does not work either (like adding identical messages to a $RULE_ID context), because every message would update the context timeout, and the contexts of high-traffic patterns would be kept indefinitely (or at least until they eat up the memory). With that said, I see a point for this use case :)
One more thought on this: could you try to increase the stats_level() global option to 3, and see if the statistics contain details for patterndb matches? It might be possible that they do (statistics is sadly underdocumented in the admin guide - my fault, sorry). If this works, then you'd only have to query the statistics every ten minutes or so, and process the needed parts with a script.
Robert
Evan. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Unix Services, University Systems, University of Victoria
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Unix Services, University Systems, University of Victoria