Hi All, I have the confusion regarding logs which are not read by the Destination programs. In case of FIFO, if the destination program which is reading the logs from syslog-ng is slow and unable to sysnchronise with the syslog-ng and there is also a limit for the fifo buffer(amount of data it holds) then where does thoses logs goes , are those logs recoverable. In other case of FIFO, if there is no destination program reading the logs from the FIFO what happens to the logs which are collected by the syslog-ng are those logs droped. How syslog-ng handles all these situations. Please help me out. Thank you all in advance, Regards, torpedo
They're dropped, lost, unrecoverable. There really is nothing else syslog-ng could do in those situations. If your data is that important (and who logs un-important stuff, anyway?) you should write it to a log file, and the parse it from there; that way, if your parser dies, you don't lose any data. On Mon, 28 Feb 2005 00:02:01 -0600, torpedo <torpedo@bluebottle.com> wrote:
I have the confusion regarding logs which are not read by the Destination programs.
In case of FIFO, if the destination program which is reading the logs from syslog-ng is slow and unable to sysnchronise with the syslog-ng and there is also a limit for the fifo buffer(amount of data it holds) then where does thoses logs goes , are those logs recoverable.
In other case of FIFO, if there is no destination program reading the logs from the FIFO what happens to the logs which are collected by the syslog-ng are those logs droped.
Thanks a lot Jay. Is there any manual mentioning all these minor details of the syslog-ng ? Thanks and Regards, torpedo Quoting Jay Guerette <jayguerette@gmail.com>:
They're dropped, lost, unrecoverable. There really is nothing else syslog-ng could do in those situations.
If your data is that important (and who logs un-important stuff, anyway?) you should write it to a log file, and the parse it from there; that way, if your parser dies, you don't lose any data.
On Mon, 28 Feb 2005 00:02:01 -0600, torpedo <torpedo@bluebottle.com> wrote:
I have the confusion regarding logs which are not read by the Destination programs.
In case of FIFO, if the destination program which is reading the logs from syslog-ng is slow and unable to sysnchronise with the syslog-ng and there is also a limit for the fifo buffer(amount of data it holds) then where does thoses logs goes , are those logs recoverable.
In other case of FIFO, if there is no destination program reading the logs from the FIFO what happens to the logs which are collected by the syslog-ng are those logs droped.
syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
I wouldn't say it's a 'detail of syslog-ng'; it's more the reality of FIFOs. That said; it IS a common problem. It seems like a great idea until the other end breaks, and you lose data. Perhaps a caution is warranted in the documentation, or at the very least an entry in the FAQ; but neither exists. On Mon, 28 Feb 2005 00:29:19 -0600, torpedo <torpedo@bluebottle.com> wrote:
Thanks a lot Jay.
Is there any manual mentioning all these minor details of the syslog-ng ?
Thanks and Regards, torpedo
Quoting Jay Guerette <jayguerette@gmail.com>:
They're dropped, lost, unrecoverable. There really is nothing else syslog-ng could do in those situations.
If your data is that important (and who logs un-important stuff, anyway?) you should write it to a log file, and the parse it from there; that way, if your parser dies, you don't lose any data.
On Mon, 28 Feb 2005 00:02:01 -0600, torpedo <torpedo@bluebottle.com> wrote:
I have the confusion regarding logs which are not read by the Destination programs.
In case of FIFO, if the destination program which is reading the logs from syslog-ng is slow and unable to sysnchronise with the syslog-ng and there is also a limit for the fifo buffer(amount of data it holds) then where does thoses logs goes , are those logs recoverable.
In other case of FIFO, if there is no destination program reading the logs from the FIFO what happens to the logs which are collected by the syslog-ng are those logs droped.
syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Mon, 2005-02-28 at 01:14 -0500, Jay Guerette wrote:
They're dropped, lost, unrecoverable. There really is nothing else syslog-ng could do in those situations.
If your data is that important (and who logs un-important stuff, anyway?) you should write it to a log file, and the parse it from there; that way, if your parser dies, you don't lose any data.
This is not necessarily true for syslog-ng 1.9.x where you can specify flags(flow-control) for log statements which means to stop writing instead of dropping messages. -- Bazsi
How exactly does that work? You can only "stop writing" for so long, before a buffer fills up and you have to drop messages... On Mon, 28 Feb 2005 09:53:16 +0100, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Mon, 2005-02-28 at 01:14 -0500, Jay Guerette wrote:
They're dropped, lost, unrecoverable. There really is nothing else syslog-ng could do in those situations.
If your data is that important (and who logs un-important stuff, anyway?) you should write it to a log file, and the parse it from there; that way, if your parser dies, you don't lose any data.
This is not necessarily true for syslog-ng 1.9.x where you can specify flags(flow-control) for log statements which means to stop writing instead of dropping messages.
On Mon, 2005-02-28 at 09:27 -0500, Jay Guerette wrote:
How exactly does that work? You can only "stop writing" for so long, before a buffer fills up and you have to drop messages...
Simple: syslog-ng stops reading. if you have flow-controlled sources (e.g. unix-stream, or TCP with the other end doing flow control as well) then at the end the application will block on sending the message.
On Mon, 28 Feb 2005 09:53:16 +0100, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Mon, 2005-02-28 at 01:14 -0500, Jay Guerette wrote:
They're dropped, lost, unrecoverable. There really is nothing else syslog-ng could do in those situations.
-- Bazsi
While a nice feature, this will only help in those cases where the source stream is managed. In traditional send-and-forget-udp syslog, this would not help the risk of data loss when the destination is a FIFO. On Mon, 28 Feb 2005 17:08:24 +0100, Balazs Scheidler <bazsi@balabit.hu> wrote:
How exactly does that work? You can only "stop writing" for so long, before a buffer fills up and you have to drop messages...
Simple: syslog-ng stops reading. if you have flow-controlled sources (e.g. unix-stream, or TCP with the other end doing flow control as well) then at the end the application will block on sending the message.
They're dropped, lost, unrecoverable. There really is nothing else syslog-ng could do in those situations.
participants (3)
-
Balazs Scheidler
-
Jay Guerette
-
torpedo