[syslog-ng] syslog CPU usage is sawtooth

[Evan Rempel]

Balazs Scheidler wrote:
> On Tue, 2007-11-13 at 13:04 -0800, Evan Rempel wrote:
>> We were running syslog-ng with default setting for time_sleep (which is 0)
>> and were consuming approx 50% of a single CPU with syslog-ng. We were concerned about
>> scalability because syslog-ng is single threaded, so we started tuning with time_sleep.
>> We set it to 5 and adjusted related parameters (log_fetch_limit(250), log_fifo_size(1000000))
>> and now we are seeing a sawtooth CPU usage with a small IO wait (2%) constant on top of the
>> "user" cpu sawtooth.
>>
>> Tooth	cpu	duration
>>
>> 1	1-10%	25 minutes
>> 2	2-10%	10 minutes
>> 3	3-10%	10 minutes
>> 4	4-10%	10 minutes
>>
>> This pattern repeats.
>>
>>
>> Any ideas on why the sawtooth pattern started appearing only after we adjusted these settings?
> 
> Could you post an mrtg image? I guess you were gathering this input from
> an image of some kind, and it's very difficult to understand without
> actually seeing the picture.
> 
> You know a picture is worth thousand words :) which applies to this case
> very nicely.
> 
> You can send it to me in private, if you think it contains sensitive
> information.
> 

Here are two images, one is of the CPU on the system, and the other is the count of the syslog messages
per minute.

Don't spend too much time on it because we have another system that is getting the same copy of
syslog messages and it does not show this pattern. The challenge for me is that the other system is a lot faster,
so it basically runs at 1% usage, and it is running Redhat 4. Additionally, the system I am asking the question about is 
routing message streams to program destinations that may be doing some kind of buffering and processing that could 
explain the cpu pattern. Watching "top" when this is going on shows that it is the primary syslog-ng process (the one 
accepting messages from the network) that is using the cpu, but I'm sill not 100% sure that there isn't some other
relationship.

The question is more of "have you seen this before?" or "is this expected behavior?".

Evan.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: cpu.png
Type: image/png
Size: 11122 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20071114/a2198c33/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rate.png
Type: image/png
Size: 24206 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20071114/a2198c33/attachment-0003.png