[syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio

Thu Nov 7 20:47:19 CET 2013

On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo at viasat.com> wrote:
>
> Thank you for your feedback.
>
>
>
> We are trying to design a log system that protects against log loss, so
if any of the failure scenarios that you mentioned occurred, we need to
make sure we do not lose any logs.  That is the reason why we want to use
an audit log file stored in non-volatile memory to store logs that have not
been sent to the server yet.  However, we are having trouble defining how
to know when to delete logs from the file when they are delivered to the
server.
>
>
>
> I have read a little about RLTP and my understanding is that this
protocol is only available for syslog-ng PE edition.  We are using
syslog-ng OSE because we need to make some minor modifications to the
source code.

If you could elaborate about the changes, the pe team might be able to work
on that to integrate to the pe edition. No promises though, but it'd be
useful to know a little more about them.

 For that reason, we cannot use the RLTP protocol.
>
>
>
> Let me ask a few specific questions:
>
>
>
> 1)    Is there any way to ask the syslog-ng client daemon to send
information back to the application that sends messages to it when a
message has been successfully delivered to the syslog-ng server daemon
(message deliver acknowledgment)?

The most reliable would probably be to hand the audit records to syslog-ng
as a file, and let syslog-ng track where it's at.

syslog-ng keeps counters how many messages are processed by a source, once
enough messages are received the files can be removed.

The current file position in the current ose release gets updated as soon
as the record is queued in the destination's memory based queue.

This solution would lose messages when syslogng crashes lacking a disk
based queue. Even we had that, it'd have to be resistant to crashes.

I'm afraid only the PE edition would solve the problem completely. I am not
here to propagate that, but your requirements are way over traditional
syslog. The PE team did a lot of work to improve reliability in various
failure scenarios effectively converting syslog-ng to a reliable queueing
implementation.

Bits and pieces are coming to the ose, but it's complex and tricky code,
that takes time for me (and the ose team) to take and integrate, and some
pieces are proprietary.

>
>
>
> 2)    I have been looking for a C/C++ API to generate IETF syslog-ng
messages but I cannot not find it.  I believe that the #include <syslog.h>
header is specific for BSD syslog messages.  If there is one, where can I
find it?

I don't know any such implementation for c++, log4j has one.

But its really not difficult to write, format the message, and post it
using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a
single send operation in your c++ app once the message is formatted.

>
>
>
> 3)    I did a little reading on ampq and zeromp, do you know if I can
save messages to non-volatile memory while they are waiting in the queue?
I need to make sure that messages are not lost if the system is turned-off
or rebooted.

These do have disk buffers, I'm not sure what happens if the client crashes.

>
>
>
> Thank you for all your help.
>
>

Hope this helps at least somewhat.
>
>
>
> From: syslog-ng-bounces at lists.balabit.hu [mailto:
syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
> Sent: Thursday, November 07, 2013 2:55 AM
> To: Syslog-ng users' and developers' mailing list
> Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and
Action
>
>
>
> It depends what failure modes you want to protect against.
>
> There can be
> * network connection breaks
> * Syslog-ng daemon crashes (client, server)
> * audit daemon crashes
>
> Syslog-ng has most of the infrastructure to carry out (flow control), but
some pieces are missing. The PE team did work in this area, they created
RLTP for app. Level network acknowledgements that would protect against
network and syslogng crashes. Even in that case though the ack information
can only be propagated back to your audit app by speaking RLTP.
>
> Maybe you would need a queueing stack like amqp or 0mq, those tend to
provide explicit acks, and there are brokerless solutions as well.
>
> On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo at viasat.com> wrote:
>
> Hello,
>
>
>
> I am new at using syslog-ng and I was hoping someone in the developers
mailing list would be able to help me.
>
>
>
> I am trying to use a syslog-ng client daemon and a syslog-ng server
daemon to implement an audit message system but I cannot find information
that would help me configure the daemons for my particular scenario.
>
>
>
> Here is my scenario:
>
>
>
> I have a separate audit daemon that generates log messages that are
written to an audit log file in syslog format.  I need the syslog-ng client
to read the logs in the audit log file and send them to the syslog-ng
server.  When the server has received the messages, I need some
acknowledgment from the syslog-ng client, so my other audit daemon can
remove the submitted log messages from the audit log file (preventing it
from reaching maximum capacity).
>
>
>
> My scenario does not have to be setup exactly this way if there are
better ways to achieve the same result.  Basically, I need to keep all logs
that have not yet being sent to the syslog-ng server in an audit file.
When the messages are delivered to the syslog-ng server, I need to delete
them from the audit file.
>
>
>
> Can someone tell me if I can achieve this results using syslog-ng
daemons?  Is there a better way to implement my scenario.
>
>
>
> Thank you for your help in advance.
>
>
>
______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20131107/2f5950b9/attachment-0001.htm 