[syslog-ng] Periodic "Destination timed out, reaping" and disconnection on a write fifo (pipe)

[Attila Szakacs (aszakacs)]

Hi Charles,

I do not think, that we should remove existing functionalities, but we can add an option to disable time-reap().

I have opened a PR, that allows the user to disable time-reap().
https://github.com/balabit/syslog-ng/pull/2798

Let's see what others think about this.

Best regards,
Attila
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Charles Eidsness <charles at ccxtechnologies.com>
Sent: Monday, June 24, 2019 1:30 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Periodic "Destination timed out, reaping" and disconnection on a write fifo (pipe)

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


Hi Attila,

Thanks for the response!

It seems like Sagan stops working when the pipe is closed and then
doesn't reopen it, so it stops working on the first closure.

Could this time-reap feature maybe be dropped for fifos? Are there
cases where you are opening multiple fifos, or is it just files?

Regards,
Charles

==
Charles Eidsness
Chief Engineer
CCX Technologies
(613)791-9715


On Mon, 24 Jun 2019 at 05:33, Attila Szakacs (aszakacs)
<Attila.Szakacs at oneidentity.com> wrote:
>
> Hi Charles,
>
> In syslog-ng you can have file, pipe, etc... destinations with templated path. This template can contain time-related values, for example the hour, minute, second when the message was sent.
> We can imagine a scenario, where we want to store the incoming messages in separate files for every minute, so we set the template accordingly. Days gone by, and we opened so many file descriptors, that we reach the kernel FD limit.
> To avoid situations like this, we introduced a global option called time-reap(). It will close the file descriptors, which were not used in a long time. You can set this timeout in seconds, and it is 60 by default.
>
> Unfortunately time-reap() has its flaws.
> The first problem: It can only be set globally, for all destinations.
> The second problem: It can not be disabled. We should be able to disable it, if we are sure, that no problematic templating is set for that destination.
> The third problem: Instead of starting/resetting the timeout when we sent out a message to that destination, the timer is started when the FD is opened/reopened. Additionally, when the timeout runs out, we check if there is any pending message to that destination, which might not be true, if we are not sending anything to that FD in that exact moment.
>
> I plan to fix the first two problems, as they are relatively easy to fix. I will consult with the team.
>
> Regarding your case, you can set time-reap to a really high value, but it still will not make sure, that your pipe will not be closed.
> To find a workaround, could you tell me how impactful this problem is for you? Does sagan work well after the FD is reopened?
>
> Best regards,
> Attila
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190624/19161d5e/attachment.html>