[syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio

Fri Nov 8 17:39:58 CET 2013

Yup, rltp supports tls.
On Nov 8, 2013 4:50 PM, "Tamayo, Andres" <Andres.Tamayo at viasat.com> wrote:

>  Thanks.
>
>
>
> I am sorry, I know I keep saying this is my last question but the more I
> learn the more questions I have.
>
>
>
> Here is one more:
>
>
>
> I am thinking about using syslog-ng PE RLTP to make sure all my messages
> are delivered to the server client.  However, I have a requirement to send
> messages encrypted using the TLS protocol.  If I use the RLTP protocol, can
> I still encrypt the messages with TLS?
>
>
>
> Please let me know, thank you.
>
>
>
>
>
> *From:* syslog-ng-bounces at lists.balabit.hu [mailto:
> syslog-ng-bounces at lists.balabit.hu] *On Behalf Of *Balazs Scheidler
> *Sent:* Friday, November 08, 2013 12:02 AM
> *To:* Syslog-ng users' and developers' mailing list
> *Subject:* Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and
> Actio
>
>
>
> Yup, definitely. Just send a HUP signal, or with recent versions run
> syslog-ng-ctl reload.
>
> On Nov 7, 2013 11:37 PM, "Tamayo, Andres" <Andres.Tamayo at viasat.com>
> wrote:
>
> Once again, thank you for your feedback.
>
>
>
> One last question: Is it possible to re-load the configuration file when
> the syslog-ng client is already running?  I understand the syslog-ng client
> reads its configuration file when it is started, what I was wondering is if
> it is possible to change its configuration at run-time.  If this is not
> possible using the configuration file, is it possible to change the client
> daemon behavior when it is already executing?
>
>
>
> Thank you.
>
>
>
>
>
> *From:* syslog-ng-bounces at lists.balabit.hu [mailto:
> syslog-ng-bounces at lists.balabit.hu] *On Behalf Of *Balazs Scheidler
> *Sent:* Thursday, November 07, 2013 12:47 PM
> *To:* Syslog-ng users' and developers' mailing list
> *Subject:* Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and
> Actio
>
>
>
>
> On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo at viasat.com> wrote:
> >
> > Thank you for your feedback.
> >
> >
> >
> > We are trying to design a log system that protects against log loss, so
> if any of the failure scenarios that you mentioned occurred, we need to
> make sure we do not lose any logs.  That is the reason why we want to use
> an audit log file stored in non-volatile memory to store logs that have not
> been sent to the server yet.  However, we are having trouble defining how
> to know when to delete logs from the file when they are delivered to the
> server.
> >
> >
> >
> > I have read a little about RLTP and my understanding is that this
> protocol is only available for syslog-ng PE edition.  We are using
> syslog-ng OSE because we need to make some minor modifications to the
> source code.
>
> If you could elaborate about the changes, the pe team might be able to
> work on that to integrate to the pe edition. No promises though, but it'd
> be useful to know a little more about them.
>
>  For that reason, we cannot use the RLTP protocol.
> >
> >
> >
> > Let me ask a few specific questions:
> >
> >
> >
> > 1)    Is there any way to ask the syslog-ng client daemon to send
> information back to the application that sends messages to it when a
> message has been successfully delivered to the syslog-ng server daemon
> (message deliver acknowledgment)?
>
> The most reliable would probably be to hand the audit records to syslog-ng
> as a file, and let syslog-ng track where it's at.
>
> syslog-ng keeps counters how many messages are processed by a source, once
> enough messages are received the files can be removed.
>
> The current file position in the current ose release gets updated as soon
> as the record is queued in the destination's memory based queue.
>
> This solution would lose messages when syslogng crashes lacking a disk
> based queue. Even we had that, it'd have to be resistant to crashes.
>
> I'm afraid only the PE edition would solve the problem completely. I am
> not here to propagate that, but your requirements are way over traditional
> syslog. The PE team did a lot of work to improve reliability in various
> failure scenarios effectively converting syslog-ng to a reliable queueing
> implementation.
>
> Bits and pieces are coming to the ose, but it's complex and tricky code,
> that takes time for me (and the ose team) to take and integrate, and some
> pieces are proprietary.
>
> >
> >
> >
> > 2)    I have been looking for a C/C++ API to generate IETF syslog-ng
> messages but I cannot not find it.  I believe that the #include <syslog.h>
> header is specific for BSD syslog messages.  If there is one, where can I
> find it?
>
> I don't know any such implementation for c++, log4j has one.
>
> But its really not difficult to write, format the message, and post it
> using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a
> single send operation in your c++ app once the message is formatted.
>
> >
> >
> >
> > 3)    I did a little reading on ampq and zeromp, do you know if I can
> save messages to non-volatile memory while they are waiting in the queue?
> I need to make sure that messages are not lost if the system is turned-off
> or rebooted.
>
> These do have disk buffers, I'm not sure what happens if the client
> crashes.
>
> >
> >
> >
> > Thank you for all your help.
> >
> >
>
> Hope this helps at least somewhat.
> >
> >
> >
> > From: syslog-ng-bounces at lists.balabit.hu [mailto:
> syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
> > Sent: Thursday, November 07, 2013 2:55 AM
> > To: Syslog-ng users' and developers' mailing list
> > Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and
> Action
> >
> >
> >
> > It depends what failure modes you want to protect against.
> >
> > There can be
> > * network connection breaks
> > * Syslog-ng daemon crashes (client, server)
> > * audit daemon crashes
> >
> > Syslog-ng has most of the infrastructure to carry out (flow control),
> but some pieces are missing. The PE team did work in this area, they
> created RLTP for app. Level network acknowledgements that would protect
> against network and syslogng crashes. Even in that case though the ack
> information can only be propagated back to your audit app by speaking RLTP.
> >
> > Maybe you would need a queueing stack like amqp or 0mq, those tend to
> provide explicit acks, and there are brokerless solutions as well.
> >
> > On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo at viasat.com>
> wrote:
> >
> > Hello,
> >
> >
> >
> > I am new at using syslog-ng and I was hoping someone in the developers
> mailing list would be able to help me.
> >
> >
> >
> > I am trying to use a syslog-ng client daemon and a syslog-ng server
> daemon to implement an audit message system but I cannot find information
> that would help me configure the daemons for my particular scenario.
> >
> >
> >
> > Here is my scenario:
> >
> >
> >
> > I have a separate audit daemon that generates log messages that are
> written to an audit log file in syslog format.  I need the syslog-ng client
> to read the logs in the audit log file and send them to the syslog-ng
> server.  When the server has received the messages, I need some
> acknowledgment from the syslog-ng client, so my other audit daemon can
> remove the submitted log messages from the audit log file (preventing it
> from reaching maximum capacity).
> >
> >
> >
> > My scenario does not have to be setup exactly this way if there are
> better ways to achieve the same result.  Basically, I need to keep all logs
> that have not yet being sent to the syslog-ng server in an audit file.
> When the messages are delivered to the syslog-ng server, I need to delete
> them from the audit file.
> >
> >
> >
> > Can someone tell me if I can achieve this results using syslog-ng
> daemons?  Is there a better way to implement my scenario.
> >
> >
> >
> > Thank you for your help in advance.
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60>
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423>
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60>
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423>
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
> >
> >
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=2a87352884c2334e0e6dbbd3b9ee3b0b38c696a8707bcad828b5f4c512019b3b>
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=91c4987eef3f9a8945f7d0d52a32c7d78b111fadb475054479c3af8b6e29cb8d>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=defbb81b5f289d646e2f6223a3acafcb32bfb04d5ed27206d2ce407feb010517>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20131108/cf9ba52f/attachment-0001.htm 