Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio
On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Thank you for your feedback.
We are trying to design a log system that protects against log loss, so
if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server.
I have read a little about RLTP and my understanding is that this
protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code. If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them. For that reason, we cannot use the RLTP protocol.
Let me ask a few specific questions:
1) Is there any way to ask the syslog-ng client daemon to send
information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)? The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at. syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed. The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue. This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes. I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation. Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary.
2) I have been looking for a C/C++ API to generate IETF syslog-ng
messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it? I don't know any such implementation for c++, log4j has one. But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted.
3) I did a little reading on ampq and zeromp, do you know if I can
save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted. These do have disk buffers, I'm not sure what happens if the client crashes.
Thank you for all your help.
Hope this helps at least somewhat.
From: syslog-ng-bounces@lists.balabit.hu [mailto:
syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Thursday, November 07, 2013 2:55 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action
It depends what failure modes you want to protect against.
There can be * network connection breaks * Syslog-ng daemon crashes (client, server) * audit daemon crashes
Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP.
Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well.
On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Hello,
I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me.
I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario.
Here is my scenario:
I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity).
My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file.
Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario.
Thank you for your help in advance.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Once again, thank you for your feedback. One last question: Is it possible to re-load the configuration file when the syslog-ng client is already running? I understand the syslog-ng client reads its configuration file when it is started, what I was wondering is if it is possible to change its configuration at run-time. If this is not possible using the configuration file, is it possible to change the client daemon behavior when it is already executing? Thank you. From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Thursday, November 07, 2013 12:47 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com<mailto:Andres.Tamayo@viasat.com>> wrote:
Thank you for your feedback.
We are trying to design a log system that protects against log loss, so if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server.
I have read a little about RLTP and my understanding is that this protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code.
If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them. For that reason, we cannot use the RLTP protocol.
Let me ask a few specific questions:
1) Is there any way to ask the syslog-ng client daemon to send information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)?
The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at. syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed. The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue. This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes. I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation. Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary.
2) I have been looking for a C/C++ API to generate IETF syslog-ng messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it?
I don't know any such implementation for c++, log4j has one. But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted.
3) I did a little reading on ampq and zeromp, do you know if I can save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted.
These do have disk buffers, I'm not sure what happens if the client crashes.
Thank you for all your help.
Hope this helps at least somewhat.
From: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>] On Behalf Of Balazs Scheidler Sent: Thursday, November 07, 2013 2:55 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action
It depends what failure modes you want to protect against.
There can be * network connection breaks * Syslog-ng daemon crashes (client, server) * audit daemon crashes
Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP.
Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well.
On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com<mailto:Andres.Tamayo@viasat.com>> wrote:
Hello,
I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me.
I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario.
Here is my scenario:
I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity).
My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file.
Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario.
Thank you for your help in advance.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
Yup, definitely. Just send a HUP signal, or with recent versions run syslog-ng-ctl reload. On Nov 7, 2013 11:37 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Once again, thank you for your feedback.
One last question: Is it possible to re-load the configuration file when the syslog-ng client is already running? I understand the syslog-ng client reads its configuration file when it is started, what I was wondering is if it is possible to change its configuration at run-time. If this is not possible using the configuration file, is it possible to change the client daemon behavior when it is already executing?
Thank you.
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Balazs Scheidler *Sent:* Thursday, November 07, 2013 12:47 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio
On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Thank you for your feedback.
We are trying to design a log system that protects against log loss, so
if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server.
I have read a little about RLTP and my understanding is that this
protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code.
If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them.
For that reason, we cannot use the RLTP protocol.
Let me ask a few specific questions:
1) Is there any way to ask the syslog-ng client daemon to send
information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)?
The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at.
syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed.
The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue.
This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes.
I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation.
Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary.
2) I have been looking for a C/C++ API to generate IETF syslog-ng
messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it?
I don't know any such implementation for c++, log4j has one.
But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted.
3) I did a little reading on ampq and zeromp, do you know if I can
save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted.
These do have disk buffers, I'm not sure what happens if the client crashes.
Thank you for all your help.
Hope this helps at least somewhat.
From: syslog-ng-bounces@lists.balabit.hu [mailto:
syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Thursday, November 07, 2013 2:55 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action
It depends what failure modes you want to protect against.
There can be * network connection breaks * Syslog-ng daemon crashes (client, server) * audit daemon crashes
Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP.
Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well.
On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Hello,
I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me.
I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario.
Here is my scenario:
I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity).
My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file.
Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario.
Thank you for your help in advance.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks. I am sorry, I know I keep saying this is my last question but the more I learn the more questions I have. Here is one more: I am thinking about using syslog-ng PE RLTP to make sure all my messages are delivered to the server client. However, I have a requirement to send messages encrypted using the TLS protocol. If I use the RLTP protocol, can I still encrypt the messages with TLS? Please let me know, thank you. From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Friday, November 08, 2013 12:02 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio Yup, definitely. Just send a HUP signal, or with recent versions run syslog-ng-ctl reload. On Nov 7, 2013 11:37 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com<mailto:Andres.Tamayo@viasat.com>> wrote: Once again, thank you for your feedback. One last question: Is it possible to re-load the configuration file when the syslog-ng client is already running? I understand the syslog-ng client reads its configuration file when it is started, what I was wondering is if it is possible to change its configuration at run-time. If this is not possible using the configuration file, is it possible to change the client daemon behavior when it is already executing? Thank you. From: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>] On Behalf Of Balazs Scheidler Sent: Thursday, November 07, 2013 12:47 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com<mailto:Andres.Tamayo@viasat.com>> wrote:
Thank you for your feedback.
We are trying to design a log system that protects against log loss, so if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server.
I have read a little about RLTP and my understanding is that this protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code.
If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them. For that reason, we cannot use the RLTP protocol.
Let me ask a few specific questions:
1) Is there any way to ask the syslog-ng client daemon to send information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)?
The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at. syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed. The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue. This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes. I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation. Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary.
2) I have been looking for a C/C++ API to generate IETF syslog-ng messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it?
I don't know any such implementation for c++, log4j has one. But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted.
3) I did a little reading on ampq and zeromp, do you know if I can save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted.
These do have disk buffers, I'm not sure what happens if the client crashes.
Thank you for all your help.
Hope this helps at least somewhat.
From: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>] On Behalf Of Balazs Scheidler Sent: Thursday, November 07, 2013 2:55 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action
It depends what failure modes you want to protect against.
There can be * network connection breaks * Syslog-ng daemon crashes (client, server) * audit daemon crashes
Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP.
Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well.
On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com<mailto:Andres.Tamayo@viasat.com>> wrote:
Hello,
I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me.
I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario.
Here is my scenario:
I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity).
My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file.
Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario.
Thank you for your help in advance.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=2a87352884c2334e0e6dbbd3b9ee3b0b38c696a8707bcad828b5f4c512019b3b> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=91c4987eef3f9a8945f7d0d52a32c7d78b111fadb475054479c3af8b6e29cb8d> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=defbb81b5f289d646e2f6223a3acafcb32bfb04d5ed27206d2ce407feb010517>
Yup, rltp supports tls. On Nov 8, 2013 4:50 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Thanks.
I am sorry, I know I keep saying this is my last question but the more I learn the more questions I have.
Here is one more:
I am thinking about using syslog-ng PE RLTP to make sure all my messages are delivered to the server client. However, I have a requirement to send messages encrypted using the TLS protocol. If I use the RLTP protocol, can I still encrypt the messages with TLS?
Please let me know, thank you.
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Balazs Scheidler *Sent:* Friday, November 08, 2013 12:02 AM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio
Yup, definitely. Just send a HUP signal, or with recent versions run syslog-ng-ctl reload.
On Nov 7, 2013 11:37 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Once again, thank you for your feedback.
One last question: Is it possible to re-load the configuration file when the syslog-ng client is already running? I understand the syslog-ng client reads its configuration file when it is started, what I was wondering is if it is possible to change its configuration at run-time. If this is not possible using the configuration file, is it possible to change the client daemon behavior when it is already executing?
Thank you.
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Balazs Scheidler *Sent:* Thursday, November 07, 2013 12:47 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio
On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Thank you for your feedback.
We are trying to design a log system that protects against log loss, so
if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server.
I have read a little about RLTP and my understanding is that this
protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code.
If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them.
For that reason, we cannot use the RLTP protocol.
Let me ask a few specific questions:
1) Is there any way to ask the syslog-ng client daemon to send
information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)?
The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at.
syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed.
The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue.
This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes.
I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation.
Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary.
2) I have been looking for a C/C++ API to generate IETF syslog-ng
messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it?
I don't know any such implementation for c++, log4j has one.
But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted.
3) I did a little reading on ampq and zeromp, do you know if I can
save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted.
These do have disk buffers, I'm not sure what happens if the client crashes.
Thank you for all your help.
Hope this helps at least somewhat.
From: syslog-ng-bounces@lists.balabit.hu [mailto:
syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Thursday, November 07, 2013 2:55 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action
It depends what failure modes you want to protect against.
There can be * network connection breaks * Syslog-ng daemon crashes (client, server) * audit daemon crashes
Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP.
Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well.
On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo@viasat.com> wrote:
Hello,
I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me.
I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario.
Here is my scenario:
I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity).
My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file.
Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario.
Thank you for your help in advance.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=a1945aa4488349b321541f219f9fd10fbdb53d8186594d73f3af3e9531e9ec60> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c500f64d1fedc40e7bd15670e042b374ae9b25d094f326b4c3d172df4ea24423> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=iEyROIc2pnaw2E6hT1a6yxnZE1omFnf69rMa4F98oUg%3D%0A&s=c98bcfdff402d7a7731ee35aa79ba6456dba0f5ea8bf2ca2b2b780fa578ed0fe>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v1/url?u=https://lists.balabit.hu/mailman/listinfo/syslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=2a87352884c2334e0e6dbbd3b9ee3b0b38c696a8707bcad828b5f4c512019b3b> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/support/documentation/?product%3Dsyslog-ng&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=91c4987eef3f9a8945f7d0d52a32c7d78b111fadb475054479c3af8b6e29cb8d> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v1/url?u=http://www.balabit.com/wiki/syslog-ng-faq&k=OWT%2FB14AE7ysJN06F7d2nQ%3D%3D%0A&r=o7GqWGDQQvuJbgvvR6T88i9N1FvBWrMPYBwWrEtosko%3D%0A&m=ev69N7c%2F5jtRzr90KvUZfH07trbMVa9vM7f3%2BLkTVKg%3D%0A&s=defbb81b5f289d646e2f6223a3acafcb32bfb04d5ed27206d2ce407feb010517>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Balazs Scheidler
-
Tamayo, Andres