syslog-ng vs (of all things) Win2k + IIS
This afternoon I was in a debate with another one of the software engineering guys here were I work. He has made the claim that for 300 machines each logging a message at least every 10 seconds (probably more like once every other second.) that encapsultaing the message in an XML tag and then sending that to a windblows 2000 box via HTTP running IIS, where it has to process the message using a CGI or ASP script, is more scalable than using syslog-ng, and that syslog-ng can't even handle this load. Now, I would beg to differ on this assesment, but I need to find some numbers to backup my claim. Can anyone help? (Ohh.. and you don't have to necessarily use an Intel box for the main server) Matthew M. Copeland Software Engineer matthew.copeland@honeywell.com
On Thu, 5 Oct 2000 matthew.copeland@honeywell.com wrote:
This afternoon I was in a debate with another one of the software engineering guys here were I work. He has made the claim that for 300 machines each logging a message at least every 10 seconds (probably more like once every other second.) that encapsultaing the message in an XML tag and then sending that to a windblows 2000 box via HTTP running IIS, where it has to process the message using a CGI or ASP script, is more scalable than using syslog-ng, and that syslog-ng can't even handle this load. Now, I would beg to differ on this assesment, but I need to find some numbers to backup my claim. Can anyone help? (Ohh.. and you don't have to necessarily use an Intel box for the main server)
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :) A rate of 150 messages per second should be really easy for syslog-ng. The other "solution" will spend more time running DOM parsers than anything else. -jwb
Jeffrey W. Baker on Thu 5/10 15:07 -0700:
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :)
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive. Syslog-ng could be more efficient still by allocating large chunks of memory (maybe using obstacks) for each destination and then batch-writing them (say, when an alarm expires). I imagine that syslog-ng spends a lot of time in system calls because it writes each message individually.
On Fri, 6 Oct 2000, Scott McDermott wrote:
Jeffrey W. Baker on Thu 5/10 15:07 -0700:
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :)
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive.
I don't know, and yes. The current syslog-ng design works like this. There is an iteration loop that syslog-ng goes through. At the beginning of the loop, it reads log entries from the source. Then it adds these log entries to a destination queue. Then it flushes the destination queue. Repeat. The problem is that if you read N entries from the input, and your queue size is only n, where N > n, N - n entries will be discarded. Unfortunately, raising the queue size (via log_fifo_size) is not the solution. If you do that, then syslog-ng's memory consumption will increase at the rate of N' - n', and eventually will either overflow the queue size anyway, or exhaust your system of memory.
Syslog-ng could be more efficient still by allocating large chunks of memory (maybe using obstacks) for each destination and then batch-writing them (say, when an alarm expires). I imagine that syslog-ng spends a lot of time in system calls because it writes each message individually.
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :)
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive.
I don't know, and yes.
The current syslog-ng design works like this. There is an iteration loop that syslog-ng goes through. At the beginning of the loop, it reads log entries from the source. Then it adds these log entries to a destination queue. Then it flushes the destination queue. Repeat.
The problem is that if you read N entries from the input, and your queue size is only n, where N > n, N - n entries will be discarded.
This is not true. Syslog-ng uses a fully nonblocking I/O backend based on poll() and callbacks. If poll detects that an fd is readable, a callback is called which reads all available data from the fd (but maximum MAX_LINE bytes, which is currently 2048) and each line in this chunk is sent towards the destination. This procedure is repeated for each fd (either to be read or written). The worst case is the following scenario: 1) the destination fd is the first to be polled, so there's nothing waiting to be written. 2) each source fd is after the destination fd, and each has some data available 3) the destination will not be flushed until the next iteration You only have to make sure that the destination fifo doesn't overflow in a single poll loop. If you assume that a 2048 byte block may contain 20 messages, and you have 10 sources, you'll have to increase the fifo size to at least 10*20=200 entries.
Unfortunately, raising the queue size (via log_fifo_size) is not the solution. If you do that, then syslog-ng's memory consumption will increase at the rate of N' - n', and eventually will either overflow the queue size anyway, or exhaust your system of memory.
see above. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
The worst case is the following scenario: 1) the destination fd is the first to be polled, so there's nothing waiting to be written. 2) each source fd is after the destination fd, and each has some data available 3) the destination will not be flushed until the next iteration
You only have to make sure that the destination fifo doesn't overflow in a single poll loop. If you assume that a 2048 byte block may contain 20 messages, and you have 10 sources, you'll have to increase the fifo size to at least 10*20=200 entries.
What are the implications of changing max_connections to 300 in afinet.c in the routine make_afinet_source? I am wanting this to be able to be used by 300 (or maybe a little more) computers for remote logging. I also need to make it so that if it loses a connection it will queue the information until it reconnects probably upto some bound. If I change the fifo length upto something much larger, is it going to cause a problem? Matthew M. Copeland
On Tue, Oct 10, 2000 at 01:43:56PM -0500, matthew.copeland@honeywell.com wrote:
The worst case is the following scenario: 1) the destination fd is the first to be polled, so there's nothing waiting to be written. 2) each source fd is after the destination fd, and each has some data available 3) the destination will not be flushed until the next iteration
You only have to make sure that the destination fifo doesn't overflow in a single poll loop. If you assume that a 2048 byte block may contain 20 messages, and you have 10 sources, you'll have to increase the fifo size to at least 10*20=200 entries.
What are the implications of changing max_connections to 300 in afinet.c in the routine make_afinet_source? I am wanting this to be able to be used by 300 (or maybe a little more) computers for remote logging.
Worst case is that each of your 300 source connections has data in it, maximum 20 messages, so you should increase your log_fifo_size to 6000. the memory usage of this should be around 6000*200*1.5 (with 200 bytes average log message and 50% overhead) that's 2megs of memory per destination.
I also need to make it so that if it loses a connection it will queue the information until it reconnects probably upto some bound. If I change the fifo length upto something much larger, is it going to cause a problem?
It's the client where you should increase the fifo value to the number of messages you wish to keep buffered. 10000 fifo entries should be enough for 10000 seconds = 166 minutes ~ 2.5 hours, assuming you have a single log message per second. This may use about 3MB memory. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
What are the implications of changing max_connections to 300 in afinet.c in the routine make_afinet_source? I am wanting this to be able to be used by 300 (or maybe a little more) computers for remote logging.
Worst case is that each of your 300 source connections has data in it, maximum 20 messages, so you should increase your log_fifo_size to 6000. the memory usage of this should be around 6000*200*1.5 (with 200 bytes average log message and 50% overhead) that's 2megs of memory per destination.
I tried changing the max_connections in afinet.c from 10 to 11, and it will no longer compile when I do this. It seems to throw errors from make_class saying that there is no file or directory. If I change it back to 10, it will still not compile. (It compiles before the change to 11.) No other changes from the standard 1.4.7. Any thoughts? Matthew M. Copeland
matthew.copeland@honeywell.com on Tue 10/10 15:41 -0500:
I tried changing the max_connections in afinet.c from 10 to 11, and it will no longer compile when I do this. It seems to throw errors from make_class saying that there is no file or directory. If I change it back to 10, it will still not compile. (It compiles before the change to 11.) No other changes from the standard 1.4.7. Any thoughts?
Yes, I ran into this myself when screwing around with the source. Here's my relevant build notes: README/syslog-ng requires libol to be installed, but links statically, so we give it --with-libol pointed at the build tree (we are the only program as of 20001004 that uses libol) note that any change to the source files seems to cause the libol class configurator (libol seems to basically be a bunch of routines for OO C programming, and comes with a preprocessor for C source files where you can define classes in the .x file or something like that) to need to be run. This of course requires scsh, see README/libol README/libol This package requires scsh (the Scheme shell from MIT) to build properly. It will appear to build without it, but some of the resulting scripts specify "#! \" as their interpreter (it's supposed to be #!/path/to/scsh but since it wasn't found...) and if they are ever called on (syslog-ng does call on them if you make changes to source files) they will fail to execute. configure like this: # important! watch configure output to be sure it detects the # scsh export PATH=$PATH:/path/to/scsh/build/tree/root/scsh ./configure README/scsh Currently only using this package to support reconfiguration of libol/syslog-ng tree. For this reason we are not installing the program but leaving it in the build directory. To do this use make libdir=`pwd` else it won't run properly. `configure' needs to be given no special arguments since we don't care where it's installed and there's not any special features to enable or disable. Also note that, although not documented, scsh requires scshvm in the same directory as itself. Simply copy buildroot/scshvm to buildroot/scsh/ Hope this helps.
I tried changing the max_connections in afinet.c from 10 to 11, and it will no longer compile when I do this. It seems to throw errors from make_class saying that there is no file or directory. If I change it back to 10, it will still not compile. (It compiles before the change to 11.) No other changes from the standard 1.4.7. Any thoughts?
max_connections is not a compile time parameter, in your syslog-ng.conf: source src { tcp(max_connections(300)); }; btw: the compilation problem is caused by a missing scsh on your system, simply touch afinet.c.x and try to remake. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
Scott McDermott (mcdermot@questra.com) wrote:
Jeffrey W. Baker on Thu 5/10 15:07 -0700:
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :)
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive.
I would imagine that syslog-ng would be able to keep up fine with not many filters and a fast machine. I have about 120 Unix/NT systems logging to syslog-ng, on the most part it does well keeping up with it but I do have a few filters which I'm sure are quite intensive considering each line logged is sent through these filters. What does it do if it can't deal with the log straight away? Puts it in the buffer for when it can deal with it? What if the buffer gets full? I would imagine that's why there is the garbage collection stuff, cause it does buffer. I could be very wrong ?! Where is Balazs. :-)
Syslog-ng could be more efficient still by allocating large chunks of memory (maybe using obstacks) for each destination and then batch-writing them (say, when an alarm expires). I imagine that syslog-ng spends a lot of time in system calls because it writes each message individually.
I didn't realise that. From what the documentation says sync() does something that might help you. John
John on Fri 6/10 16:21 +0100:
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive.
I would imagine that syslog-ng would be able to keep up fine with not many filters and a fast machine. I have about 120 Unix/NT systems logging to syslog-ng, on the most part it does well keeping up with it but I do have a few filters which I'm sure are quite intensive considering each line logged is sent through these filters. What does it do if it can't deal with the log straight away? Puts it in the buffer for when it can deal with it? What if the buffer gets full?
Then it reallocs more space :) The only thing that should cause it to lose messages is an OOM event. You just keep writing to a big buffer for the destination, then have it flushed periodically. If the disk can't keep up, I guess you are screwed, but it should at least try, and increase the buffer size if necessary. Then it should be able to write out the buffer better during periods when it isn't as swamped with incoming messages. Currently there seems to be a hard limit on the buffer size and this surely will lead to tragedy.
I would imagine that's why there is the garbage collection stuff, cause it does buffer. I could be very wrong ?! Where is Balazs. :-)
Syslog-ng could be more efficient still by allocating large chunks of memory (maybe using obstacks) for each destination and then batch-writing them (say, when an alarm expires). I imagine that syslog-ng spends a lot of time in system calls because it writes each message individually.
I didn't realise that.
Well, it seems I was only partially right. The logging subsystem actually does do it using a FIFO queue, as Jeff Baker points out. I am not sure how often the queue is flushed (every loop? what's the point of that?), but it does seem that when it *is* flushed to the IO subsystem, each message is written out individually (strace/truss reveals this) to the descriptor. These should be batched, IMO, so perhaps dozens or hundreds of messages are written out in one write(2); this would allow syslog-ng to keep up with a much larger message stream. A variable buffer for each FD, and possibly an alarm per destination, or even a separate metathread for flushing might be one way to go.
From what the documentation says sync() does something that might help you.
No, this just determines how often sync() is called, which flushes all file descriptors before it returns (or at least schedules them for flushing...the semantics depend on OS flavor as to the result of the flush when sync(2) returns). What I want to do is be able to schedule the writes themselves, not how often we force the writes to be committed by the kernel.
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :)
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive.
I would imagine that syslog-ng would be able to keep up fine with not many filters and a fast machine. I have about 120 Unix/NT systems logging to syslog-ng, on the most part it does well keeping up with it but I do have a few filters which I'm sure are quite intensive considering each line logged is sent through these filters. What does it do if it can't deal with the log straight away? Puts it in the buffer for when it can deal with it? What if the buffer gets full?
I would imagine that's why there is the garbage collection stuff, cause it does buffer. I could be very wrong ?! Where is Balazs. :-)
syslog-ng reads messages, and processes them, if the load is high processing them takes longer, so it has no time reading its input file descriptors. If it doesn't read those input fds, the kernel begins to drop messages (in case of UDP, in case of TCP, windows are not acknowledged)
Syslog-ng could be more efficient still by allocating large chunks of memory (maybe using obstacks) for each destination and then batch-writing them (say, when an alarm expires). I imagine that syslog-ng spends a lot of time in system calls because it writes each message individually.
I didn't realise that. From what the documentation says sync() does something that might help you.
sync doesn't batch writes, it simply controls the number of lines collected in the buffer before _anything_ is written. So if you have sync(10) 10 lines is collected, and then each of them is written individually. So this could be improved. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :)
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive.
You can control the size of the output buffer with the log_fifo_size() option. Of course this size is not preallocated, it's just the maximum number of entries to be buffered. The default value is 100.
Syslog-ng could be more efficient still by allocating large chunks of memory (maybe using obstacks) for each destination and then batch-writing them (say, when an alarm expires). I imagine that syslog-ng spends a lot of time in system calls because it writes each message individually.
Yes, this may be a place for improvement. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
This afternoon I was in a debate with another one of the software engineering guys here were I work. He has made the claim that for 300 machines each logging a message at least every 10 seconds (probably more like once every other second.) that encapsultaing the message in an XML tag and then sending that to a windblows 2000 box via HTTP running IIS, where it has to process the message using a CGI or ASP script, is more scalable than using syslog-ng, and that syslog-ng can't even handle this load. Now, I would beg to differ on this assesment, but I need to find some numbers to backup my claim. Can anyone help? (Ohh.. and you don't have to necessarily use an Intel box for the main server)
Matthew M. Copeland Software Engineer matthew.copeland@honeywell.com
ROFLMAO! He REALLY thinks a Win2k box in THAT configuration can outdo syslog-ng on a good UNIX boxen for logging? (snicker, giggle). OOOOOkkkaayyy... I don't have any hard numbers, but I've tested it with about 20 remote boxes doing a "while true do;logger blah blah ; done" loop, and syslog-ng never broke a good sweat (although the standard syslogd gets it's lunch ate by such loads; which was what I was testing: syslog-ng vs syslogd. Syslog-ng was so far and away the hands down winner that I didn't even bother to record the hard numbers). If I get time tomorrow, I'll see if I can rack up some hard numbers for ya. Maybe even whip up a little C code to really blast the hell out of it. Anyhoo... that's my 2 cents. -- A.L.Lambert
That would be great. The big thing they seem to be harping on is that using TCP over udp in the syslog will make it much slower, since we have to use TCP for the transmissions. I am assuming that someone here will know this. When you use tcp logging for remote syslog-ng, does it keep the tcp connection open, or does it initiate a new connection each time a message is posted? Matthew M. Copeland
ROFLMAO! He REALLY thinks a Win2k box in THAT configuration can outdo syslog-ng on a good UNIX boxen for logging? (snicker, giggle). OOOOOkkkaayyy... I don't have any hard numbers, but I've tested it with about 20 remote boxes doing a "while true do;logger blah blah ; done" loop, and syslog-ng never broke a good sweat (although the standard syslogd gets it's lunch ate by such loads; which was what I was testing: syslog-ng vs syslogd. Syslog-ng was so far and away the hands down winner that I didn't even bother to record the hard numbers). If I get time tomorrow, I'll see if I can rack up some hard numbers for ya. Maybe even whip up a little C code to really blast the hell out of it. Anyhoo... that's my 2 cents.
-- A.L.Lambert
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng
matthew.copeland@honeywell.com on Fri 6/10 08:34 -0500:
That would be great. The big thing they seem to be harping on is that using TCP over udp in the syslog will make it much slower, since we have to use TCP for the transmissions.
For system logs, I'll take slowness over lack of reliability any day. Sure, if your network people have their shit together, you can rest with a pretty good idea that you won't have any UDP packets dropped on your own networks. Still, that's not a guarantee, which TCP gives. But try routing from your WAN sites with UDP, or worse, from remote VPN sites that have to route over the Internet. TCP is a big win here if you want all your log packets. Why the original UNIX syslog started with UDP is beyond my comprehension. Here we have logs which may or may not be *critical* in the case of intrusion attempts or other problems where missing log messages would be a disaster. And unless you are running at modem speeds or something (in which case you'd *have* to use TCP anyways or you'd have tons of lost messages), who cares about the additional overhead of TCP...this isn't NFS we're talking about; we're not at the races. We're trying to peice together what went wrong.
I am assuming that someone here will know this. When you use tcp logging for remote syslog-ng, does it keep the tcp connection open, or does it initiate a new connection each time a message is posted?
Yes, it does keep the connection open until it detects that the remote has closed the connection, and then it will re-connect.
On Fri, Oct 06, 2000 at 10:05:55AM -0400, Scott McDermott wrote:
Sure, if your network people have their shit together, you can rest with a pretty good idea that you won't have any UDP packets dropped on your own networks. Still, that's not a guarantee, which TCP gives.
TCP gives a conditional guarantee. Your packets only arrive safely if the network isn't too congested and if the hosts have the time to establish a full session. That sucks when you want to send a "network congested" message or a "my power supply is dying so I'm going down" message or a "my CPU is at 100% and i'm dropping packets left and right" message. You need a protocol that works when everything around it is breaking. The younger SNMP-trap is also UDP.
Why the original UNIX syslog started with UDP is beyond my comprehension. Here we have logs which may or may not be *critical* in the case of intrusion attempts or other problems where missing log messages would be a disaster.
Because the people who wrote syslog lived in the days before everyone was concerned with security and secure loghosts, when they were just trying to make everything work to begin with. And because they wanted a protocol that had some chance of working during other kinds of disasters, like imminent host failure. And because they (and the people who wrote snmp, and snmp-trap, and DNS) were looking at tiny messages, at TCP overhead that's considerably larger than the underlying messages, at TCP's long session setup/teardown times, and so at much higher scalability limits for UDP over TCP for a given collection of hardware and network. - Morty
participants (7)
-
A.L.Lambert
-
Balazs Scheidler
-
Jeffrey W. Baker
-
John
-
matthew.copeland＠honeywell.com
-
Mordechai T. Abzug
-
Scott McDermott