Missing one log message when destination unavailable for short period of time.
Hi all, I tried to setup simple relay with syslog-ng, it receives log on one port and forward to the destination. It works fine when destination is available, but when destination unavailable for a short period of time and then became available I miss one log entry. For example assume that I send simple numbers with logger command to syslog-ng, from 0 to 100, when I receive 50 on destination I stop its listener and start it again, in this case the first log entry that I receive is 52 instead of 51. Here is my syslog-ng.conf: @version: 3.26 options { time_reopen(5); }; source s_network { network( ip("127.0.0.1") port(514) transport("udp") ); }; destination d_network { network( "127.0.0.1" port(5514) transport("udp") ); }; log { source(s_network); destination(d_network); }; Would you please help me to resolve this issue? Regards, Ali
The tcp stack of the host would return success for the first message that follows a tcp connection termination. We have no way of knowing that it was actually sent or not, the host kernel doesn't return this information. Syslog-ng tries to detect connection termination proactively and as long as we don't have a message to write we should detect the closing connection and avoid the loss of the upcoming message. Please check if syslog-ng detects the closing connection, it should report this in its debug log level, look for something like "eof detected on an idle connection". The complete solution to this is application level acknowledgement, which is an extra protocol layer over tcp. Syslog-ng PE has rltp for this purpose but that's a proprietary feature and I found its extra overhead is rarely needed. Bazsi On Thu, May 7, 2020, 19:03 Ali Shirvani <aj.shirvani@gmail.com> wrote:
Hi all,
I tried to setup simple relay with syslog-ng, it receives log on one port and forward to the destination. It works fine when destination is available, but when destination unavailable for a short period of time and then became available I miss one log entry.
For example assume that I send simple numbers with logger command to syslog-ng, from 0 to 100, when I receive 50 on destination I stop its listener and start it again, in this case the first log entry that I receive is 52 instead of 51.
Here is my syslog-ng.conf:
@version: 3.26
options { time_reopen(5); };
source s_network { network( ip("127.0.0.1") port(514) transport("udp") ); };
destination d_network { network( "127.0.0.1" port(5514) transport("udp") ); };
log { source(s_network); destination(d_network); };
Would you please help me to resolve this issue?
Regards, Ali
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks for your reply, I has been checked the syslog-ng debug console, it detected the broken connection, and queue further messages and send them when the link is available. But I think it doesn't detect link down properly. On Thu, May 7, 2020 at 10:15 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
The tcp stack of the host would return success for the first message that follows a tcp connection termination.
We have no way of knowing that it was actually sent or not, the host kernel doesn't return this information.
Syslog-ng tries to detect connection termination proactively and as long as we don't have a message to write we should detect the closing connection and avoid the loss of the upcoming message. Please check if syslog-ng detects the closing connection, it should report this in its debug log level, look for something like "eof detected on an idle connection".
The complete solution to this is application level acknowledgement, which is an extra protocol layer over tcp.
Syslog-ng PE has rltp for this purpose but that's a proprietary feature and I found its extra overhead is rarely needed.
Bazsi
On Thu, May 7, 2020, 19:03 Ali Shirvani <aj.shirvani@gmail.com> wrote:
Hi all,
I tried to setup simple relay with syslog-ng, it receives log on one port and forward to the destination. It works fine when destination is available, but when destination unavailable for a short period of time and then became available I miss one log entry.
For example assume that I send simple numbers with logger command to syslog-ng, from 0 to 100, when I receive 50 on destination I stop its listener and start it again, in this case the first log entry that I receive is 52 instead of 51.
Here is my syslog-ng.conf:
@version: 3.26
options { time_reopen(5); };
source s_network { network( ip("127.0.0.1") port(514) transport("udp") ); };
destination d_network { network( "127.0.0.1" port(5514) transport("udp") ); };
log { source(s_network); destination(d_network); };
Would you please help me to resolve this issue?
Regards, Ali
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I should note that, I tried TLS destination with disk-buffer and reliable flag, but I have no success. Still missing one log entry after short downtime. On Thu, May 7, 2020 at 10:20 PM Ali Shirvani <aj.shirvani@gmail.com> wrote:
Thanks for your reply, I has been checked the syslog-ng debug console, it detected the broken connection, and queue further messages and send them when the link is available. But I think it doesn't detect link down properly.
On Thu, May 7, 2020 at 10:15 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
The tcp stack of the host would return success for the first message that follows a tcp connection termination.
We have no way of knowing that it was actually sent or not, the host kernel doesn't return this information.
Syslog-ng tries to detect connection termination proactively and as long as we don't have a message to write we should detect the closing connection and avoid the loss of the upcoming message. Please check if syslog-ng detects the closing connection, it should report this in its debug log level, look for something like "eof detected on an idle connection".
The complete solution to this is application level acknowledgement, which is an extra protocol layer over tcp.
Syslog-ng PE has rltp for this purpose but that's a proprietary feature and I found its extra overhead is rarely needed.
Bazsi
On Thu, May 7, 2020, 19:03 Ali Shirvani <aj.shirvani@gmail.com> wrote:
Hi all,
I tried to setup simple relay with syslog-ng, it receives log on one port and forward to the destination. It works fine when destination is available, but when destination unavailable for a short period of time and then became available I miss one log entry.
For example assume that I send simple numbers with logger command to syslog-ng, from 0 to 100, when I receive 50 on destination I stop its listener and start it again, in this case the first log entry that I receive is 52 instead of 51.
Here is my syslog-ng.conf:
@version: 3.26
options { time_reopen(5); };
source s_network { network( ip("127.0.0.1") port(514) transport("udp") ); };
destination d_network { network( "127.0.0.1" port(5514) transport("udp") ); };
log { source(s_network); destination(d_network); };
Would you please help me to resolve this issue?
Regards, Ali
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
If you think that's the case, please submit a bug report with details why you think it is behaving incorrectly. On Thu, May 7, 2020, 19:51 Ali Shirvani <aj.shirvani@gmail.com> wrote:
Thanks for your reply, I has been checked the syslog-ng debug console, it detected the broken connection, and queue further messages and send them when the link is available. But I think it doesn't detect link down properly.
On Thu, May 7, 2020 at 10:15 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
The tcp stack of the host would return success for the first message that follows a tcp connection termination.
We have no way of knowing that it was actually sent or not, the host kernel doesn't return this information.
Syslog-ng tries to detect connection termination proactively and as long as we don't have a message to write we should detect the closing connection and avoid the loss of the upcoming message. Please check if syslog-ng detects the closing connection, it should report this in its debug log level, look for something like "eof detected on an idle connection".
The complete solution to this is application level acknowledgement, which is an extra protocol layer over tcp.
Syslog-ng PE has rltp for this purpose but that's a proprietary feature and I found its extra overhead is rarely needed.
Bazsi
On Thu, May 7, 2020, 19:03 Ali Shirvani <aj.shirvani@gmail.com> wrote:
Hi all,
I tried to setup simple relay with syslog-ng, it receives log on one port and forward to the destination. It works fine when destination is available, but when destination unavailable for a short period of time and then became available I miss one log entry.
For example assume that I send simple numbers with logger command to syslog-ng, from 0 to 100, when I receive 50 on destination I stop its listener and start it again, in this case the first log entry that I receive is 52 instead of 51.
Here is my syslog-ng.conf:
@version: 3.26
options { time_reopen(5); };
source s_network { network( ip("127.0.0.1") port(514) transport("udp") ); };
destination d_network { network( "127.0.0.1" port(5514) transport("udp") ); };
log { source(s_network); destination(d_network); };
Would you please help me to resolve this issue?
Regards, Ali
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks for your guidance, I'm using syslog-ng docker file, should I test with bare syslog-ng without docker before filling an issue? On Thu, May 7, 2020 at 10:45 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
If you think that's the case, please submit a bug report with details why you think it is behaving incorrectly.
On Thu, May 7, 2020, 19:51 Ali Shirvani <aj.shirvani@gmail.com> wrote:
Thanks for your reply, I has been checked the syslog-ng debug console, it detected the broken connection, and queue further messages and send them when the link is available. But I think it doesn't detect link down properly.
On Thu, May 7, 2020 at 10:15 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
The tcp stack of the host would return success for the first message that follows a tcp connection termination.
We have no way of knowing that it was actually sent or not, the host kernel doesn't return this information.
Syslog-ng tries to detect connection termination proactively and as long as we don't have a message to write we should detect the closing connection and avoid the loss of the upcoming message. Please check if syslog-ng detects the closing connection, it should report this in its debug log level, look for something like "eof detected on an idle connection".
The complete solution to this is application level acknowledgement, which is an extra protocol layer over tcp.
Syslog-ng PE has rltp for this purpose but that's a proprietary feature and I found its extra overhead is rarely needed.
Bazsi
On Thu, May 7, 2020, 19:03 Ali Shirvani <aj.shirvani@gmail.com> wrote:
Hi all,
I tried to setup simple relay with syslog-ng, it receives log on one port and forward to the destination. It works fine when destination is available, but when destination unavailable for a short period of time and then became available I miss one log entry.
For example assume that I send simple numbers with logger command to syslog-ng, from 0 to 100, when I receive 50 on destination I stop its listener and start it again, in this case the first log entry that I receive is 52 instead of 51.
Here is my syslog-ng.conf:
@version: 3.26
options { time_reopen(5); };
source s_network { network( ip("127.0.0.1") port(514) transport("udp") ); };
destination d_network { network( "127.0.0.1" port(5514) transport("udp") ); };
log { source(s_network); destination(d_network); };
Would you please help me to resolve this issue?
Regards, Ali
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi,
Thanks for your guidance, I'm using syslog-ng docker file, should I test with bare syslog-ng without docker before filling an issue?
It may come in handy if we know whether you can reproduce the issue without Docker's network interfaces. Please also share the exact reproduction steps in the issue. -- László Várady
participants (3)
-
Ali Shirvani
-
Balazs Scheidler
-
László Várady (lvarady)