Balazs Scheidler wrote:
On Tue, 2007-11-13 at 13:04 -0800, Evan Rempel wrote:
We were running syslog-ng with default setting for time_sleep (which is 0) and were consuming approx 50% of a single CPU with syslog-ng. We were concerned about scalability because syslog-ng is single threaded, so we started tuning with time_sleep. We set it to 5 and adjusted related parameters (log_fetch_limit(250), log_fifo_size(1000000)) and now we are seeing a sawtooth CPU usage with a small IO wait (2%) constant on top of the "user" cpu sawtooth.
Tooth cpu duration
1 1-10% 25 minutes 2 2-10% 10 minutes 3 3-10% 10 minutes 4 4-10% 10 minutes
This pattern repeats.
Any ideas on why the sawtooth pattern started appearing only after we adjusted these settings?
Could you post an mrtg image? I guess you were gathering this input from an image of some kind, and it's very difficult to understand without actually seeing the picture.
You know a picture is worth thousand words :) which applies to this case very nicely.
You can send it to me in private, if you think it contains sensitive information.
Here are two images, one is of the CPU on the system, and the other is the count of the syslog messages per minute. Don't spend too much time on it because we have another system that is getting the same copy of syslog messages and it does not show this pattern. The challenge for me is that the other system is a lot faster, so it basically runs at 1% usage, and it is running Redhat 4. Additionally, the system I am asking the question about is routing message streams to program destinations that may be doing some kind of buffering and processing that could explain the cpu pattern. Watching "top" when this is going on shows that it is the primary syslog-ng process (the one accepting messages from the network) that is using the cpu, but I'm sill not 100% sure that there isn't some other relationship. The question is more of "have you seen this before?" or "is this expected behavior?". Evan.
