Hi, Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server? Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space. Thank you for your help! --Vlad
"Malkin, Vlad" <Vlad.Malkin@viasat.com> writes:
Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server?
No, it's not possible to do that, but read below.
Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space.
This is something that syslog-ng PE can do for you (albeit still not allowing you to use an external program for ACKing) by way of its RLTP protocol and disk buffering. Other than that, using something like AMQP with durability and persistance on would be closest to what you want, but that still allows a narrow window where messages can get lost. -- |8]
Thank you for responding! I'm not as much concerned with the reliability on-the-wire, TCP is good enough to detect most network issues. There is a loose assumption that if the message gets to the stack on the server side, the server will be able to get the data, i.e. it in itself had not failed. But leaving all that aside, what I'm really after is when syslog-ng client sends a message and it "believes" that it has been successfully sent, how could a call back to another process be inserted? Should hard flow control be used between message producer and the syslog-ng, I'm assuming that the client will stop reading from the producer if it is unable to send, correct? If we wanted to insert code in syslog-ng to call another process upon successful message send, do you know where that would need to be? Thank you! --Vlad -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Monday, November 18, 2013 3:49 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question. "Malkin, Vlad" <Vlad.Malkin@viasat.com> writes:
Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server?
No, it's not possible to do that, but read below.
Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space.
This is something that syslog-ng PE can do for you (albeit still not allowing you to use an external program for ACKing) by way of its RLTP protocol and disk buffering. Other than that, using something like AMQP with durability and persistance on would be closest to what you want, but that still allows a narrow window where messages can get lost. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Mon, 2013-11-18 at 16:00 +0000, Malkin, Vlad wrote:
Thank you for responding!
I'm not as much concerned with the reliability on-the-wire, TCP is good enough to detect most network issues. There is a loose assumption that if the message gets to the stack on the server side, the server will be able to get the data, i.e. it in itself had not failed. But leaving all that aside, what I'm really after is when syslog-ng client sends a message and it "believes" that it has been successfully sent, how could a call back to another process be inserted? Should hard flow control be used between message producer and the syslog-ng, I'm assuming that the client will stop reading from the producer if it is unable to send, correct?
If we wanted to insert code in syslog-ng to call another process upon successful message send, do you know where that would need to be?
in the in-process acknowledgement path. log_source_msg_ack() is the final ACK callback that increments the window and awakens the souce if it went to sleep because of the lack of window space. But calling out to an external process is resource hungry, syslog-ng is able to process hundreds of thousands of messages per second, with a fork/exec like that I'd guess it wouldn't be more than a couple of thousands.
Thank you! --Vlad
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Monday, November 18, 2013 3:49 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question.
"Malkin, Vlad" <Vlad.Malkin@viasat.com> writes:
Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server?
No, it's not possible to do that, but read below.
Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space.
This is something that syslog-ng PE can do for you (albeit still not allowing you to use an external program for ACKing) by way of its RLTP protocol and disk buffering.
Other than that, using something like AMQP with durability and persistance on would be closest to what you want, but that still allows a narrow window where messages can get lost.
Balazs, Thank you very much! --Vlad -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Tuesday, November 19, 2013 5:24 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question. On Mon, 2013-11-18 at 16:00 +0000, Malkin, Vlad wrote:
Thank you for responding!
I'm not as much concerned with the reliability on-the-wire, TCP is good enough to detect most network issues. There is a loose assumption that if the message gets to the stack on the server side, the server will be able to get the data, i.e. it in itself had not failed. But leaving all that aside, what I'm really after is when syslog-ng client sends a message and it "believes" that it has been successfully sent, how could a call back to another process be inserted? Should hard flow control be used between message producer and the syslog-ng, I'm assuming that the client will stop reading from the producer if it is unable to send, correct?
If we wanted to insert code in syslog-ng to call another process upon successful message send, do you know where that would need to be?
in the in-process acknowledgement path. log_source_msg_ack() is the final ACK callback that increments the window and awakens the souce if it went to sleep because of the lack of window space. But calling out to an external process is resource hungry, syslog-ng is able to process hundreds of thousands of messages per second, with a fork/exec like that I'd guess it wouldn't be more than a couple of thousands.
Thank you! --Vlad
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Monday, November 18, 2013 3:49 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question.
"Malkin, Vlad" <Vlad.Malkin@viasat.com> writes:
Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server?
No, it's not possible to do that, but read below.
Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space.
This is something that syslog-ng PE can do for you (albeit still not allowing you to use an external program for ACKing) by way of its RLTP protocol and disk buffering.
Other than that, using something like AMQP with durability and persistance on would be closest to what you want, but that still allows a narrow window where messages can get lost.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You could also look at the apache FLUME project. It has reliable transport and could accept data from syslog-ng, although it can not read data directly from the syslog kernel channels, and there would still not be any verification that t=syslog-ng got its message to fluke. The likelihood would be that it would sinde syslog-ng and FLUKE could run on the same host. *https://flume*.*apache*.org Evan. On 11/25/2013 03:19 PM, Malkin, Vlad wrote:
Balazs,
Thank you very much!
--Vlad
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Tuesday, November 19, 2013 5:24 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question.
On Mon, 2013-11-18 at 16:00 +0000, Malkin, Vlad wrote:
Thank you for responding!
I'm not as much concerned with the reliability on-the-wire, TCP is good enough to detect most network issues. There is a loose assumption that if the message gets to the stack on the server side, the server will be able to get the data, i.e. it in itself had not failed. But leaving all that aside, what I'm really after is when syslog-ng client sends a message and it "believes" that it has been successfully sent, how could a call back to another process be inserted? Should hard flow control be used between message producer and the syslog-ng, I'm assuming that the client will stop reading from the producer if it is unable to send, correct?
If we wanted to insert code in syslog-ng to call another process upon successful message send, do you know where that would need to be? in the in-process acknowledgement path.
log_source_msg_ack() is the final ACK callback that increments the window and awakens the souce if it went to sleep because of the lack of window space.
But calling out to an external process is resource hungry, syslog-ng is able to process hundreds of thousands of messages per second, with a fork/exec like that I'd guess it wouldn't be more than a couple of thousands.
Thank you! --Vlad
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Monday, November 18, 2013 3:49 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question.
"Malkin, Vlad" <Vlad.Malkin@viasat.com> writes:
Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server? No, it's not possible to do that, but read below.
Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space. This is something that syslog-ng PE can do for you (albeit still not allowing you to use an external program for ACKing) by way of its RLTP protocol and disk buffering.
Other than that, using something like AMQP with durability and persistance on would be closest to what you want, but that still allows a narrow window where messages can get lost.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hello Balazs, Thank you for your previous feedback about where to insert code to send message delivery acknowledgment to an external process. I have been trying to understand the code in the log_source_msg_ack() function. You pointed out this function as the final ACK callback. Unfortunately I have not been able to really understand the code and I am not sure if this function is called in multiple threads or not. Basically, I want to make sure that this function is called when a syslog-ng client message was been received by the syslog-ng server. Is this the case? Or is this function called when a syslog-ng client message has been sent to the syslog-ng server? If that is the case, we would like to use the TCP protocol to check if the message has arrived to the syslog-ng server. Which syslog-ng function is in charge of sending messages over TCP to the syslog-ng server? Thank you for your help. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Tuesday, November 19, 2013 5:24 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question. On Mon, 2013-11-18 at 16:00 +0000, Malkin, Vlad wrote:
Thank you for responding!
I'm not as much concerned with the reliability on-the-wire, TCP is good enough to detect most network issues. There is a loose assumption that if the message gets to the stack on the server side, the server will be able to get the data, i.e. it in itself had not failed. But leaving all that aside, what I'm really after is when syslog-ng client sends a message and it "believes" that it has been successfully sent, how could a call back to another process be inserted? Should hard flow control be used between message producer and the syslog-ng, I'm assuming that the client will stop reading from the producer if it is unable to send, correct?
If we wanted to insert code in syslog-ng to call another process upon successful message send, do you know where that would need to be?
in the in-process acknowledgement path. log_source_msg_ack() is the final ACK callback that increments the window and awakens the souce if it went to sleep because of the lack of window space. But calling out to an external process is resource hungry, syslog-ng is able to process hundreds of thousands of messages per second, with a fork/exec like that I'd guess it wouldn't be more than a couple of thousands.
Thank you! --Vlad
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Monday, November 18, 2013 3:49 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng OSE Question.
"Malkin, Vlad" <Vlad.Malkin@viasat.com> writes:
Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server?
No, it's not possible to do that, but read below.
Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space.
This is something that syslog-ng PE can do for you (albeit still not allowing you to use an external program for ACKing) by way of its RLTP protocol and disk buffering.
Other than that, using something like AMQP with durability and persistance on would be closest to what you want, but that still allows a narrow window where messages can get lost.
______________________________________________________________________________ Member info: https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/... Documentation: https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/do... FAQ: https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslo...
participants (5)
-
Balazs Scheidler
-
Evan Rempel
-
Gergely Nagy
-
Malkin, Vlad
-
Tamayo, Andres