[syslog-ng] syslog-ng Message Deliver Acknowledgment and Action

Thu Nov 7 16:09:24 CET 2013

Thank you for your feedback.

We are trying to design a log system that protects against log loss, so if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs.  That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet.  However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server.

I have read a little about RLTP and my understanding is that this protocol is only available for syslog-ng PE edition.  We are using syslog-ng OSE because we need to make some minor modifications to the source code.  For that reason, we cannot use the RLTP protocol.

Let me ask a few specific questions:

1)    Is there any way to ask the syslog-ng client daemon to send information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)?

2)    I have been looking for a C/C++ API to generate IETF syslog-ng messages but I cannot not find it.  I believe that the #include <syslog.h> header is specific for BSD syslog messages.  If there is one, where can I find it?

3)    I did a little reading on ampq and zeromp, do you know if I can save messages to non-volatile memory while they are waiting in the queue?  I need to make sure that messages are not lost if the system is turned-off or rebooted.

Thank you for all your help.

From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Thursday, November 07, 2013 2:55 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action

It depends what failure modes you want to protect against.

There can be
* network connection breaks
* Syslog-ng daemon crashes (client, server)
* audit daemon crashes

Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP.

Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well.
On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo at viasat.com<mailto:Andres.Tamayo at viasat.com>> wrote:
Hello,

I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me.

I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario.

Here is my scenario:

I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format.  I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server.  When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity).

My scenario does not have to be setup exactly this way if there are better ways to achieve the same result.  Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file.  When the messages are delivered to the syslog-ng server, I need to delete them from the audit file.

Can someone tell me if I can achieve this results using syslog-ng daemons?  Is there a better way to implement my scenario.

Thank you for your help in advance.

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=H3XFCCxuN%2BoJ7bteSxs7fSMtVHpZVOxK491YyGxJqtc%3D%0A&s=762f719282e6c489261af35e7c48511cb1e28ce0f26ba244dc1cd239a6105c65>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=H3XFCCxuN%2BoJ7bteSxs7fSMtVHpZVOxK491YyGxJqtc%3D%0A&s=816e8cd9b9abcde4d3a7344e432dbaefa88b470a09d3f585ecb9fd9fb2f2abcd>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=H3XFCCxuN%2BoJ7bteSxs7fSMtVHpZVOxK491YyGxJqtc%3D%0A&s=438420559bf0bc11020087736b3878af9ad9e723b2f78502d6bdabe3f21e9d2a>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20131107/8a986b00/attachment.htm 