[syslog-ng] Syslog-ng OSE Question.

[Tamayo, Andres]

Hello Balazs,

Thank you for your previous feedback about where to insert code to send message delivery acknowledgment to an external process.

I have been trying to understand the code in the log_source_msg_ack() function.  You pointed out this function as the final ACK callback.  Unfortunately I have not been able to really understand the code and I am not sure if this function is called in multiple threads or not.  Basically, I want to make sure that this function is called when a syslog-ng client message was been received by the syslog-ng server.  Is this the case?  

Or is this function called when a syslog-ng client message has been sent to the syslog-ng server?  If that is the case, we would like to use the TCP protocol to check if the message has arrived to the syslog-ng server.  Which syslog-ng function is in charge of sending messages over TCP to the syslog-ng server?

Thank you for your help.



-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Tuesday, November 19, 2013 5:24 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Syslog-ng OSE Question.

On Mon, 2013-11-18 at 16:00 +0000, Malkin, Vlad wrote:
> Thank you for responding!
> 
> I'm not as much concerned with the reliability on-the-wire, TCP is good enough to detect most network issues.  There is a loose assumption that if the message gets to the stack on the server side, the server will be able to get the data, i.e. it in itself had not failed.  But leaving all that aside, what I'm really after is when syslog-ng client sends a message and it "believes" that it has been successfully sent, how could a call back to another process be inserted?  Should hard flow control be used between message producer and the syslog-ng, I'm assuming that the client will stop reading from the producer if it is unable to send, correct?
> 
> If we wanted to insert code in syslog-ng to call another process upon successful message send, do you know where that would need to be?

in the in-process acknowledgement path.

log_source_msg_ack() is the final ACK callback that increments the window and awakens the souce if it went to sleep because of the lack of window space.

But calling out to an external process is resource hungry, syslog-ng is able to process hundreds of thousands of messages per second, with a fork/exec like that I'd guess it wouldn't be more than a couple of thousands.

> 
> Thank you!
> --Vlad
> 
> -----Original Message-----
> From: syslog-ng-bounces at lists.balabit.hu 
> [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Gergely Nagy
> Sent: Monday, November 18, 2013 3:49 AM
> To: Syslog-ng users' and developers' mailing list
> Subject: Re: [syslog-ng] Syslog-ng OSE Question.
> 
> "Malkin, Vlad" <Vlad.Malkin at viasat.com> writes:
> 
> > Is it possible to have syslog-ng client to invoke an external (to 
> > it) software to acknowledge when each individual log statement is 
> > sent to the server?
> 
> No, it's not possible to do that, but read below.
> 
> > Background: logs should not be lost. The system needs to store logs 
> > internally until they are transmitted to the server (use of TCP 
> > transport provides a level of reliable delivery). After each log 
> > statement is transmitted by syslog-ng client, it needs to be removed 
> > from internal store to preserve allocated space.
> 
> This is something that syslog-ng PE can do for you (albeit still not allowing you to use an external program for ACKing) by way of its RLTP protocol and disk buffering.
> 
> Other than that, using something like AMQP with durability and persistance on would be closest to what you want, but that still allows a narrow window where messages can get lost.
> 


______________________________________________________________________________
Member info: https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=yTimoFMWu61turCXtlrQlcllUocKYmjWlZTP5QsMDYw%3D%0A&s=cc48862d81dadc5cfc6a96e725ea1e054a2b0c9e51a1883eb985724ac48399b8
Documentation: https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=yTimoFMWu61turCXtlrQlcllUocKYmjWlZTP5QsMDYw%3D%0A&s=db8ae8b8e309562cf16a0df5e4258ca53e49aa7f266dd6ca482d47d9168b6f77
FAQ: https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=yTimoFMWu61turCXtlrQlcllUocKYmjWlZTP5QsMDYw%3D%0A&s=942fe9d7e74cbc4d1cf0cfe81c75ccf46b9f19079786e48d650d2757a4c86590