Disk based queuing with rsyslog?
Hi, I read that syslog-ng can be setup to use disk based with flow control with the client to prevent lost of logs. However I was just thinking, how does this mechanism works? Does the client have to be using syslog-ng? Will it work if the client is using rsyslog? Native appliance that purely support syslog? Yours Sincerely, Delon Lee
Hello Delon Lee! You don't need the client to be syslog-ng for *disk based buffering on the host*, i.e. you only need syslog-ng where you would like to do the buffering. It should work with rsyslog as a client and syslog-ng as a host. Regards, Gabor On Mon, May 7, 2018 at 5:02 PM, Delon Lee Di Lun <lee.delon2005@gmail.com> wrote:
Hi,
I read that syslog-ng can be setup to use disk based with flow control with the client to prevent lost of logs.
However I was just thinking, how does this mechanism works? Does the client have to be using syslog-ng? Will it work if the client is using rsyslog? Native appliance that purely support syslog?
Yours Sincerely, Delon Lee
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi! Let me clarify. The purpose of disk based buffering is to prevent the lost of logs due to what factors? It does prevent lost of logs due to restarting the syslog-ng right? Buffering is done on the syslog-ng server end right? Yours Sincerely, Delon Lee On Mon, 7 May 2018 at 23:18 Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hello Delon Lee!
You don't need the client to be syslog-ng for *disk based buffering on the host*, i.e. you only need syslog-ng where you would like to do the buffering. It should work with rsyslog as a client and syslog-ng as a host.
Regards, Gabor
On Mon, May 7, 2018 at 5:02 PM, Delon Lee Di Lun <lee.delon2005@gmail.com> wrote:
Hi,
I read that syslog-ng can be setup to use disk based with flow control with the client to prevent lost of logs.
However I was just thinking, how does this mechanism works? Does the client have to be using syslog-ng? Will it work if the client is using rsyslog? Native appliance that purely support syslog?
Yours Sincerely, Delon Lee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Disk buffering means you have a persistent buffer, so you can safely stop syslog-ng without worry about loosing not sent log messages (not like with the default memory buffer). Buffering can be done anywhere and makes sense to do it everywhere where a connection can be broken (network hosts, etc.). If you use the reliable disk-buffer then log loss caused by a syslog-ng crash is prevented. I would recommend you the documentation of syslog-ng, it has more details: https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n... Regards, Gabor On Tue, May 8, 2018 at 3:50 AM, Delon Lee Di Lun <lee.delon2005@gmail.com> wrote:
Hi!
Let me clarify. The purpose of disk based buffering is to prevent the lost of logs due to what factors? It does prevent lost of logs due to restarting the syslog-ng right?
Buffering is done on the syslog-ng server end right?
Yours Sincerely, Delon Lee
On Mon, 7 May 2018 at 23:18 Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hello Delon Lee!
You don't need the client to be syslog-ng for *disk based buffering on the host*, i.e. you only need syslog-ng where you would like to do the buffering. It should work with rsyslog as a client and syslog-ng as a host.
Regards, Gabor
On Mon, May 7, 2018 at 5:02 PM, Delon Lee Di Lun <lee.delon2005@gmail.com
wrote:
Hi,
I read that syslog-ng can be setup to use disk based with flow control with the client to prevent lost of logs.
However I was just thinking, how does this mechanism works? Does the client have to be using syslog-ng? Will it work if the client is using rsyslog? Native appliance that purely support syslog?
Yours Sincerely, Delon Lee
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Arh. I see. So its a setting I need to enable on the client syslog-ng configuration file. I was wonder what magical thing syslog-ng employ that automatically does the buffering when its enabled on the server side of configuration file. However, now. I see the documentation mention that flow control can be used in conjunction with disk based buffering. How does flow control plays a part? Flow control only works within the syslog-ng application? From the source to dest? Or it has the capability to detect the flow rate from the remote syslog-ng client and can "command" the syslog-ng client to slow down if the syslog-ng server cannot handle the load? Yours Sincerely, Delon Lee On Tue, 8 May 2018 at 15:32 Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Disk buffering means you have a persistent buffer, so you can safely stop syslog-ng without worry about loosing not sent log messages (not like with the default memory buffer). Buffering can be done anywhere and makes sense to do it everywhere where a connection can be broken (network hosts, etc.). If you use the reliable disk-buffer then log loss caused by a syslog-ng crash is prevented.
I would recommend you the documentation of syslog-ng, it has more details:
https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n...
Regards, Gabor
On Tue, May 8, 2018 at 3:50 AM, Delon Lee Di Lun <lee.delon2005@gmail.com> wrote:
Hi!
Let me clarify. The purpose of disk based buffering is to prevent the lost of logs due to what factors? It does prevent lost of logs due to restarting the syslog-ng right?
Buffering is done on the syslog-ng server end right?
Yours Sincerely, Delon Lee
On Mon, 7 May 2018 at 23:18 Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hello Delon Lee!
You don't need the client to be syslog-ng for *disk based buffering on the host*, i.e. you only need syslog-ng where you would like to do the buffering. It should work with rsyslog as a client and syslog-ng as a host.
Regards, Gabor
On Mon, May 7, 2018 at 5:02 PM, Delon Lee Di Lun < lee.delon2005@gmail.com> wrote:
Hi,
I read that syslog-ng can be setup to use disk based with flow control with the client to prevent lost of logs.
However I was just thinking, how does this mechanism works? Does the client have to be using syslog-ng? Will it work if the client is using rsyslog? Native appliance that purely support syslog?
Yours Sincerely, Delon Lee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> However, now. I see the documentation mention that flow control can be used Delon> in conjunction with disk based buffering. How does flow control plays a Delon> part? The documentation has a chapter on flow-control, which explains the purpose, and the behaviour of it: https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n... Delon> Flow control only works within the syslog-ng application? From the source Delon> to dest? Or it has the capability to detect the flow rate from the remote Delon> syslog-ng client and can "command" the syslog-ng client to slow down if the Delon> syslog-ng server cannot handle the load? Yes, it only works within a single syslog-ng instance. But if you have syslog-ng on both client and server, if flow-control kicks in on the server, and it slows down accepting messages, then flow-control will also kick in on the client (because it detects that messages are not going out fast enough), and it will start buffering (to disk too, if so configured). This way, you'll only lose messages if all queues fill up. -- |8]
Hi, "Gergely Nagy" <algernon@balabit.com> írta 2018-05-09 12:35-kor:
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> However, now. I see the documentation mention that flow control can be used Delon> in conjunction with disk based buffering. How does flow control plays a Delon> part?
The documentation has a chapter on flow-control, which explains the purpose, and the behaviour of it: https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n...
Delon> Flow control only works within the syslog-ng application? From the source Delon> to dest? Or it has the capability to detect the flow rate from the remote Delon> syslog-ng client and can "command" the syslog-ng client to slow down if the Delon> syslog-ng server cannot handle the load?
Yes, it only works within a single syslog-ng instance. But if you have syslog-ng on both client and server, if flow-control kicks in on the server, and it slows down accepting messages, then flow-control will also kick in on the client (because it detects that messages are not going out fast enough), and it will start buffering (to disk too, if so configured). This way, you'll only lose messages if all queues fill up.
One extra thought to extend algernon's comment: to work this properly, you must use tcp. If you use udp, the client will not have proper feedback if the server couldn't process a message. In that case kernel's udp buffer just silently drop the message. In case of tcp: if the server process doesn't read (fast enough) from the socket, then the underlying kernel won't send back the tcp acknowledments. Cheers, Gyu
Hi all, Thank you for the time in commenting. I understand that this is a syslog-ng mailling list but however, in my setup I am testing. I have to use rsyslog for the client. Has anybody tried a client as rsyslog and the server as syslog-ng does the flow-control still kick in? Assume transportation protocol is TCP. I want to see if anybody tried it before, or let me know its impossible before i commit my time into testing the setup. My intention is to leverage as much of the new features in syslog-ng as I am testing the new setup. Need the setup to "sell". Yours Sincerely, Delon Lee On Wed, 9 May 2018 at 20:01 PÁSZTOR György <pasztor@linux.gyakg.u-szeged.hu> wrote:
Hi,
"Gergely Nagy" <algernon@balabit.com> írta 2018-05-09 12:35-kor:
> "Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> However, now. I see the documentation mention that flow control can be used Delon> in conjunction with disk based buffering. How does flow control plays a Delon> part?
The documentation has a chapter on flow-control, which explains the purpose, and the behaviour of it:
https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n...
Delon> Flow control only works within the syslog-ng application?
From the source
Delon> to dest? Or it has the capability to detect the flow rate
from the remote
Delon> syslog-ng client and can "command" the syslog-ng client to
slow down if the
Delon> syslog-ng server cannot handle the load?
Yes, it only works within a single syslog-ng instance. But if you have syslog-ng on both client and server, if flow-control kicks in on the server, and it slows down accepting messages, then flow-control will also kick in on the client (because it detects that messages are not going out fast enough), and it will start buffering (to disk too, if so configured). This way, you'll only lose messages if all queues fill up.
One extra thought to extend algernon's comment: to work this properly, you must use tcp. If you use udp, the client will not have proper feedback if the server couldn't process a message. In that case kernel's udp buffer just silently drop the message. In case of tcp: if the server process doesn't read (fast enough) from the socket, then the underlying kernel won't send back the tcp acknowledments.
Cheers, Gyu
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, "Delon Lee Di Lun" <lee.delon2005@gmail.com> írta 2018-05-09 12:21-kor:
I understand that this is a syslog-ng mailling list but however, in my setup I am testing. I have to use rsyslog for the client.
Has anybody tried a client as rsyslog and the server as syslog-ng does the flow-control still kick in? Assume transportation protocol is TCP.
I want to see if anybody tried it before, or let me know its impossible before i commit my time into testing the setup.
Well, I don't have first hands experience with this setup. But based on my current knowledge: it should work. So if you set up proper flow control on the syslog-ng side for the tcp source, and you also setup a disk buffer on the sender rsyslog side, it should fulfill your expectations, and should not loose any log events. I didn't analyzed exceptional cases here, eg. when you have some kind of transparent "loadbalancer" between your rsyslog and syslog-ng, and if your rsyslog get the ack but syslog-ng didn't really received the data. It's just playing with thoughts. So I could imagine situations when all your intention aiming failsafe log transportation will let you down. But it needs a really f..d up network configuration! In this case, if you can not trust your network setup, I suggest using an extra ssl layer for mutualy check parties via certificates. syslog-ng can handle that natively: https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n... The last time I check this part of rsyslog, they suggested using an stunnel for that scenario. If it's still the case, then stunnel can be a weak point: when rsyslog thinks it handed over the data to stunnel, but if that "crashes" or restarts, then it will loose its own buffer. Honestly, I don't know if rsyslog has any improvment on that side. Cheers, Gyu
Hi, Thanks for the prompt reply. I think I got what I needed. The reason I am looking into buffering an flow control is so that the setup is resilient to maintenance reboots, for example. On Wed, 9 May 2018 at 20:40 PÁSZTOR György <pasztor@linux.gyakg.u-szeged.hu> wrote:
Hi,
"Delon Lee Di Lun" <lee.delon2005@gmail.com> írta 2018-05-09 12:21-kor:
I understand that this is a syslog-ng mailling list but however, in my setup I am testing. I have to use rsyslog for the client.
Has anybody tried a client as rsyslog and the server as syslog-ng does the flow-control still kick in? Assume transportation protocol is TCP.
I want to see if anybody tried it before, or let me know its impossible before i commit my time into testing the setup.
Well, I don't have first hands experience with this setup. But based on my current knowledge: it should work. So if you set up proper flow control on the syslog-ng side for the tcp source, and you also setup a disk buffer on the sender rsyslog side, it should fulfill your expectations, and should not loose any log events.
I didn't analyzed exceptional cases here, eg. when you have some kind of transparent "loadbalancer" between your rsyslog and syslog-ng, and if your rsyslog get the ack but syslog-ng didn't really received the data. It's just playing with thoughts. So I could imagine situations when all your intention aiming failsafe log transportation will let you down. But it needs a really f..d up network configuration! In this case, if you can not trust your network setup, I suggest using an extra ssl layer for mutualy check parties via certificates. syslog-ng can handle that natively:
https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n... The last time I check this part of rsyslog, they suggested using an stunnel for that scenario. If it's still the case, then stunnel can be a weak point: when rsyslog thinks it handed over the data to stunnel, but if that "crashes" or restarts, then it will loose its own buffer. Honestly, I don't know if rsyslog has any improvment on that side.
Cheers, Gyu
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Delon Lee Di Lun
-
Gergely Nagy
-
Nagy, Gábor
-
PÁSZTOR György