Hi there Can someone explain to me how the tcp/udp destination options handle network outages/congestion? If I set my fifo to 10000, does that mean if the remote server is down (I assume this only affects TCP - not UDP BTW), the "client" will hold up to 10000 records before dropping... the oldest? Just what does happen then? Also, does anyone think adding a compress option to the tcp destination option is a good idea? It could really cut down on the traffic (esp if you could force a *minimum* fifo queue size - to increase your chance of repeated data). It could make centralized logging over WANs much more attractive. Thanks Jason
Jason Haar wrote:
Can someone explain to me how the tcp/udp destination options handle network outages/congestion? If I set my fifo to 10000, does that mean if the remote server is down (I assume this only affects TCP - not UDP BTW), the "client" will hold up to 10000 records before dropping... the oldest? Just what does happen then?
As fas as I know, the messages are dropped silently.
Also, does anyone think adding a compress option to the tcp destination option is a good idea? It could really cut down on the traffic (esp if you could force a *minimum* fifo queue size - to increase your chance of repeated data). It could make centralized logging over WANs much more attractive.
I think compression isn't the factor for centralized logging in WAN environments. Imagine your normal messages size is approx. <512Bytes. How many messages you must create/send to flood a 64KB/128KB leased line? The important factor in centralized logging is in most cases the perfomance of the center (the log server). Compression will decrease log servers performance. We are logging >5000 devices with >15.000.000 messages per day to a centralized syslog-ng server over WAN. Problems we observed are mostly on the central size. The WAN isn't really the bottleneck in most scenarios. -- Best regards --Andreas Schulze [phone: +49.5246.80.1275, fax: +49.5246.80.2275] | I believe, it was Dennis Ritchie who said something like: | "C is rarely the best language for a given task, | but it's often the second-best". | The implication being that: "[...]" | http://www.ioccc.org/1990/dds.c
Jason Haar wrote:
Can someone explain to me how the tcp/udp destination options handle network outages/congestion? If I set my fifo to 10000, does that mean if the remote server is down (I assume this only affects TCP - not UDP BTW), the "client" will hold up to 10000 records before dropping... the oldest? Just what does happen then?
As fas as I know, the messages are dropped silently.
Also, does anyone think adding a compress option to the tcp destination option is a good idea? It could really cut down on the traffic (esp if you could force a *minimum* fifo queue size - to increase your chance of repeated data). It could make centralized logging over WANs much more attractive.
I think compression isn't the factor for centralized logging in WAN environments. But maybe its nice to have. Imagine your normal messages size is approx. <512Bytes. How many messages you must create/send to flood a 64KB/128KB leased line? The important factor in centralized logging is in most cases the perfomance of the center (the log server). Compression will decrease log servers performance. We are logging >5000 devices with >15.000.000 messages per day to a centralized syslog-ng server over WAN. Problems we observed are mostly on the central size. The WAN isn't really the bottleneck in most scenarios. -- Best regards --Andreas Schulze [phone: +49.5246.80.1275, fax: +49.5246.80.2275] | I believe, it was Dennis Ritchie who said something like: | "C is rarely the best language for a given task, | but it's often the second-best". | The implication being that: "[...]" | http://www.ioccc.org/1990/dds.c
On Tue, Mar 18, 2003 at 11:41:28AM +0100, Andreas Schulze wrote:
We are logging >5000 devices with >15.000.000 messages per day to a centralized syslog-ng server over WAN. Problems we observed are mostly on the central size. The WAN isn't really the bottleneck in most scenarios.
Andreas, Your experience in this area would be very useful to others. Would you care to expand on the problems you encountered and how you solved them? -- Nate Campi http://www.campin.net
Nate Campi wrote:
On Tue, Mar 18, 2003 at 11:41:28AM +0100, Andreas Schulze wrote:
We are logging >5000 devices with >15.000.000 messages per day to a centralized syslog-ng server over WAN. Problems we observed are mostly on the central size. The WAN isn't really the bottleneck in most scenarios.
Your experience in this area would be very useful to others. Would you care to expand on the problems you encountered and how you solved them?
Of course. But give me a little bit more time. Maybe, to make it easier for me, ask me in detail in what area you have special interest. Ad hoc, I remembering a real 'problem' we solved is the problem of feeding large messages to syslog-ng locally via syslog(3). Many libc's (Solaris, Linux) are broken here. So don't use syslog(3) locally to feed msgs >1k (approx.) to syslog-ng. A solution for e.g. a trap receiver was discussed in: > Re: [syslog-ng]Filtering Large Syslog Messages > Date: 03.02.2003 17:04 A more generic statement is: If you like to proccess tons of msgs per hour, never use filesystems for storage. Do it in memory (e.g. pipes, etc.). Filesystems are slow. Here message loss occurs. Memory is much more faster. At this point, in heavy loaded environments, you have to process the msgs at least as fast as you receive them. Of course, if you only would store the msgs and no more processing is needed, fast disk subsys are the common way. -- Best regards --Andreas Schulze [phone: +49.5246.80.1275, fax: +49.5246.80.2275] | I believe, it was Dennis Ritchie who said something like: | "C is rarely the best language for a given task, | but it's often the second-best". | The implication being that: "[...]" | http://www.ioccc.org/1990/dds.c
Andreas Schulze wrote:
I think compression isn't the factor for centralized logging in WAN environments. But maybe its nice to have. Imagine your normal messages size is approx. <512Bytes. How many messages you must create/send to flood a 64KB/128KB leased line?
Well I've managed to flood a T1 with syslog traffic from one host before...
We are logging >5000 devices with >15.000.000 messages per day to a centralized syslog-ng server over WAN. Problems we observed are mostly on the central size. The WAN isn't really the bottleneck in most scenarios.
That's really encouraging to hear. My "issue" with WAN congestion was due to an extreme condition caused by one application pouring out syslog messages - enough to flood a T1. I was burnt so badly by it that I've been reluctant to turn WAN-based logging back on again - maybe I shouldn't be... Using TCP instead of UDP would also limit the damage - Slammer showed us all how much better UDP is than TCP at filling pipes... Jason
participants (3)
-
Andreas Schulze
-
Jason Haar
-
Nate Campi