Reliability of Remote logging via TCP/Syslog-NG
Hi Guys, (New to this list, searched the FAQ for this qn, didnt find it. So I apologize if this was repeated before on the list. It might be related to the topic on TCP timingout though as report @ https://lists.balabit.hu/pipermail/syslog-ng/2005-February/006974.html) I'm planning to deploy a syslog-ng setup. (Central logging server in Asia, and Clients will be in US/HK). The setup is actually as a backup/secure repository of log entries of the appliation we are running. Now the question is, (I know TCP is reliable vs UDP), how reliable will syslog-ng hand the log entries ? If it gets disconnected (due to timeout, reset, etc), will it hold the log entry in memory and queue it for resend ? Is there an option to set before it times out (And at least logs it in a local log that there was an error (as an internal msg i guess) I need to be certain that no messages get lost basically :P Thanks in advance. Elvin -
On Wed, 2005-06-15 at 03:14 -0700, Alf wrote:
Hi Guys,
(New to this list, searched the FAQ for this qn, didnt find it. So I apologize if this was repeated before on the list. It might be related to the topic on TCP timingout though as report @ https://lists.balabit.hu/pipermail/syslog-ng/2005-February/006974.html)
I'm planning to deploy a syslog-ng setup. (Central logging server in Asia, and Clients will be in US/HK). The setup is actually as a backup/secure repository of log entries of the appliation we are running.
Now the question is, (I know TCP is reliable vs UDP), how reliable will syslog-ng hand the log entries ? If it gets disconnected (due to timeout, reset, etc), will it hold the log entry in memory and queue it for resend ? Is there an option to set before it times out (And at least logs it in a local log that there was an error (as an internal msg i guess)
syslog-ng has a FIFO in memory where it stores messages. The contents of this fifo is not persistent (as it is in memory), thus if you stop and restart syslog-ng the contents of this fifo is lost. Another thing to note is that if this FIFO becomes full, syslog-ng will start dropping messages (which it reports using the STATS message). you can manually control the fifo size using log_fifo_size() global and per-destination option. -- Bazsi
Hi Balas and guys, --- Balazs Scheidler <bazsi@balabit.hu> wrote:
syslog-ng has a FIFO in memory where it stores messages. The contents of this fifo is not persistent (as it is in memory), thus if you stop and restart syslog-ng the contents of this fifo is lost. Another thing to note is that if this FIFO becomes full, syslog-ng will start dropping messages (which it reports using the STATS message). So assuming FIFO doesnt become full, I can assume messages that failed to send will be requeued to be sent ?
you can manually control the fifo size using log_fifo_size() global and per-destination option. Does it affect perforamnce greatly if I increase the size ? (Not that I will, since my traffic will be low), or will it be just a memory usage hit thats all ?
Elvin -
On Wed, 2005-06-15 at 03:59 -0700, Alf wrote:
Hi Balas and guys,
--- Balazs Scheidler <bazsi@balabit.hu> wrote:
syslog-ng has a FIFO in memory where it stores messages. The contents of this fifo is not persistent (as it is in memory), thus if you stop and restart syslog-ng the contents of this fifo is lost. Another thing to note is that if this FIFO becomes full, syslog-ng will start dropping messages (which it reports using the STATS message). So assuming FIFO doesnt become full, I can assume messages that failed to send will be requeued to be sent ?
One other problem is that whenever the server shuts down the connection the client drops a _single_ log message. This is not easy to fix, at least not in the 1.6.x branch. See the mailing list archives for more information.
you can manually control the fifo size using log_fifo_size() global and per-destination option. Does it affect perforamnce greatly if I increase the size ? (Not that I will, since my traffic will be low), or will it be just a memory usage hit thats all ?
it only increases memory usage. -- Bazsi
On Wed, Jun 15, 2005 at 01:24:14PM +0200, Balazs Scheidler wrote:
One other problem is that whenever the server shuts down the connection the client drops a _single_ log message. This is not easy to fix, at least not in the 1.6.x branch. See the mailing list archives for more information.
I think we can work around it. Have syslog-ng send a one byte dummy message when a connection is brought up, that a receiving syslog-ng will always discard. I had to do something like this in a performance testing suite I wrote, and I see no reason syslog-ng itself can't do it. Sound reasonable? -- Nate "Man is the only animal that blushes. Or needs to." - Samuel Clemens
On Wed, 2005-06-15 at 06:23 -0700, Nate Campi wrote:
On Wed, Jun 15, 2005 at 01:24:14PM +0200, Balazs Scheidler wrote:
One other problem is that whenever the server shuts down the connection the client drops a _single_ log message. This is not easy to fix, at least not in the 1.6.x branch. See the mailing list archives for more information.
I think we can work around it. Have syslog-ng send a one byte dummy message when a connection is brought up, that a receiving syslog-ng will always discard. I had to do something like this in a performance testing suite I wrote, and I see no reason syslog-ng itself can't do it.
Sound reasonable?
Sorry, but no. It is not the first message that gets lost, its the last on the old connection. The kernel happily acks the sent message to syslog-ng, but in the end it does not actually send it. So there is no way to determine if that message was actually sent or not. The only solution I see now is what is implemented in the 1.9.x series: in addition to write the TCP socket, also read for possible EOF marks. This way syslog-ng can realize that a connection is closed. There is a small window of opportunity the same happening, but the window is small. (e.g. it is possible that the connection breaks, syslog-ng sends a message and the discover that the connection was broken) but it is still way better than the current solution. An interesting question whether the same happens on other IP stacks, or it is only Linux which shows this behaviour. (because if this is the case, the Linux kernel could be fixed as well) -- Bazsi
--On Wednesday, June 15, 2005 5:06 PM +0200 Balazs Scheidler <bazsi@balabit.hu> wrote:
Sorry, but no. It is not the first message that gets lost, its the last on the old connection. The kernel happily acks the sent message to syslog-ng, but in the end it does not actually send it. So there is no way to determine if that message was actually sent or not.
The only solution I see now is what is implemented in the 1.9.x series: in addition to write the TCP socket, also read for possible EOF marks. This way syslog-ng can realize that a connection is closed. There is a small window of opportunity the same happening, but the window is small. (e.g. it is possible that the connection breaks, syslog-ng sends a message and the discover that the connection was broken) but it is still way better than the current solution.
An interesting question whether the same happens on other IP stacks, or it is only Linux which shows this behaviour. (because if this is the case, the Linux kernel could be fixed as well)
There is rather simple option - keep the last n messages in your FIFO, and resend them if a connection is broken and later re-established. As for it being a kernel problem, I don't think so. TCP writes aren't synchronous. You'd see the same problems writing to disk - your write() is fine, but the next write(), or close(), may return an error if an I/O error had occurred (which is why you always have to check the return code from close(), boys and girls). A simple example: - remote process dies with a SIGSEGV (thus not sending FIN) - syslog-ng calls write() - kernel returns OK - kernel transmits TCP packet - kernel gets RST - syslog-ng calls write() - kernel returns EPIPE There is nothing the kernel can do, unless your IP stack supports synchronous TCP writes (where write doesn't return until the kernel receives an ACK). I'm not sure if there's a standard for this - it's been too long since I was that deep in the API. -- Carson
On Wed, 2005-06-15 at 19:00 -0400, Carson Gaspar wrote:
--On Wednesday, June 15, 2005 5:06 PM +0200 Balazs Scheidler <bazsi@balabit.hu> wrote:
There is rather simple option - keep the last n messages in your FIFO, and resend them if a connection is broken and later re-established.
As for it being a kernel problem, I don't think so. TCP writes aren't synchronous. You'd see the same problems writing to disk - your write() is fine, but the next write(), or close(), may return an error if an I/O error had occurred (which is why you always have to check the return code from close(), boys and girls). A simple example:
- remote process dies with a SIGSEGV (thus not sending FIN) - syslog-ng calls write() - kernel returns OK - kernel transmits TCP packet - kernel gets RST - syslog-ng calls write() - kernel returns EPIPE
There is nothing the kernel can do, unless your IP stack supports synchronous TCP writes (where write doesn't return until the kernel receives an ACK). I'm not sure if there's a standard for this - it's been too long since I was that deep in the API.
The problem that the same thing happens when the remote side does send something that terminates the connection. But the kernel does not react to it, but I agree there might other information pending in the kernel socket buffer. The real solution is to have some kind of application layer ack, or as you say, retransmit the last couple of messages when the connection is broken. (but how much and what happens to duplicate packets?) -- Bazsi
--On Thursday, June 16, 2005 11:25 AM +0200 Balazs Scheidler <bazsi@balabit.hu> wrote:
The problem that the same thing happens when the remote side does send something that terminates the connection. But the kernel does not react to it, but I agree there might other information pending in the kernel socket buffer. The real solution is to have some kind of application layer ack, or as you say, retransmit the last couple of messages when the connection is broken. (but how much and what happens to duplicate packets?)
I suggest a config tunable number of messages to retransmit, with the default being zero. Folks have a choice of: - losing messages (under some, if not all, disconnect circumstances) - risking duplicate messages - running code that implements syslog-reliable (or whatever we're calling it these days - my IETF syslog WG mailing list mailbox is woefully behind...) -- Carson
On Wed, 15 Jun 2005, Balazs Scheidler wrote:
On Wed, 2005-06-15 at 03:59 -0700, Alf wrote:
Hi Balas and guys,
--- Balazs Scheidler <bazsi@balabit.hu> wrote:
syslog-ng has a FIFO in memory where it stores messages. The contents of this fifo is not persistent (as it is in memory), thus if you stop and restart syslog-ng the contents of this fifo is lost. Another thing to note is that if this FIFO becomes full, syslog-ng will start dropping messages (which it reports using the STATS message). So assuming FIFO doesnt become full, I can assume messages that failed to send will be requeued to be sent ?
One other problem is that whenever the server shuts down the connection the client drops a _single_ log message. This is not easy to fix, at least not in the 1.6.x branch. See the mailing list archives for more information.
Hi! When I read that correctly, this means that on the client machine a single line of data is thrown away, before it will reach the network interface? I am also doing some syslog setup, where some embedded machines (running syslog-ng 1.5.17) will log to a central machine, also running syslog-ng 1.6.x, and I wonder, that, wenn on the client machine there is no data waiting, what happens to the first line arriving, when the connection is up again before this line arrives? I assume it shall be transmitted unharmed, or is there anything to worry about? With kind regards Olaf Hoyer -- Olaf Hoyer ohoyer@ohoyer.de Fuerchterliche Erlebniss geben zu raten, ob der, welcher sie erlebt, nicht etwas Fuerchterliches ist. (Nietzsche, Jenseits von Gut und Boese)
On Wed, 2005-06-15 at 16:45 +0200, Olaf Hoyer wrote:
On Wed, 15 Jun 2005, Balazs Scheidler wrote:
Hi!
When I read that correctly, this means that on the client machine a single line of data is thrown away, before it will reach the network interface?
Yes, exactly. syslog-ng issues a write() call, the kernel reports no problems, thus syslog-ng happily proceeds. Then on the next write() call, the kernel reports EPIPE as it should have on the first occasion.
I am also doing some syslog setup, where some embedded machines (running syslog-ng 1.5.17) will log to a central machine, also running syslog-ng 1.6.x, and I wonder, that, wenn on the client machine there is no data waiting, what happens to the first line arriving, when the connection is up again before this line arrives? I assume it shall be transmitted unharmed, or is there anything to worry about?
On the connection I mean the TCP connection and not IP connectivity. If the TCP connection breaks for whatever reason (RST, IP outage, anything), the contents of the kernel's socket buffer is lost (e.g. the data syslog-ng has assumed to be sent). This is at least one message, as the kernel does not indicate write failure on the first write after the connection was broken. A simple solution is to "watch" for broken TCP connections by poll() -ing them for reading (where EOF is guaranteed to happen immediately after the TCP connection is broken). This is implemented in 1.9.x. The real solution would be to revamp syslog over TCP, or use some of the newly designed syslog protocols (over BEEP, but that's another matter). The point is to have some kind of application layer acknowledgement of messages which would ensure that a given message was properly received by the peer and not lost in the socket buffers. This way, only messages already acknowledged by the peer would be discarded. -- Bazsi
participants (5)
-
Alf
-
Balazs Scheidler
-
Carson Gaspar
-
Nate Campi
-
Olaf Hoyer