[syslog-ng]buffer limitations and TCP compression

Thu, 20 Mar 2003 18:38:29 +0100

Nate Campi wrote:
> On Tue, Mar 18, 2003 at 11:41:28AM +0100, Andreas Schulze wrote:
> 
>>We are logging >5000 devices with >15.000.000 messages per day
>>to a centralized syslog-ng server over WAN.
>>Problems we observed are mostly on the central size. The WAN
>>isn't really the bottleneck in most scenarios.
> 
> Your experience in this area would be very useful to others. Would you
> care to expand on the problems you encountered and how you solved them?

Of course. But give me a little bit more time.

Maybe, to make it easier for me, ask me in detail in what area you
have special interest.

Ad hoc, I remembering a real 'problem' we solved is the problem
of feeding large messages to syslog-ng locally via syslog(3).
Many libc's (Solaris, Linux) are broken here.
So don't use syslog(3) locally to feed msgs >1k (approx.) to syslog-ng.
A solution for e.g. a trap receiver was discussed in:
	> Re: [syslog-ng]Filtering Large Syslog Messages
	> Date: 03.02.2003 17:04

A more generic statement is:
If you like to proccess tons of msgs per hour, never use filesystems
for storage. Do it in memory (e.g. pipes, etc.).
Filesystems are slow. Here message loss occurs.
Memory is much more faster.
At this point, in heavy loaded environments, you have to process the
msgs at least as fast as you receive them.

Of course, if you only would store the msgs and no more processing is
needed, fast disk subsys are the common way.

-- 
Best regards --Andreas Schulze
                [phone: +49.5246.80.1275, fax: +49.5246.80.2275]