[syslog-ng] Periodic "Destination timed out, reaping" and disconnection on a write fifo (pipe)

[Attila Szakacs (aszakacs)]

Hi Charles,

In syslog-ng you can have file, pipe, etc... destinations with templated path. This template can contain time-related values, for example the hour, minute, second when the message was sent.
We can imagine a scenario, where we want to store the incoming messages in separate files for every minute, so we set the template accordingly. Days gone by, and we opened so many file descriptors, that we reach the kernel FD limit.
To avoid situations like this, we introduced a global option called time-reap(). It will close the file descriptors, which were not used in a long time. You can set this timeout in seconds, and it is 60 by default.

Unfortunately time-reap() has its flaws.
The first problem: It can only be set globally, for all destinations.
The second problem: It can not be disabled. We should be able to disable it, if we are sure, that no problematic templating is set for that destination.
The third problem: Instead of starting/resetting the timeout when we sent out a message to that destination, the timer is started when the FD is opened/reopened. Additionally, when the timeout runs out, we check if there is any pending message to that destination, which might not be true, if we are not sending anything to that FD in that exact moment.

I plan to fix the first two problems, as they are relatively easy to fix. I will consult with the team.

Regarding your case, you can set time-reap to a really high value, but it still will not make sure, that your pipe will not be closed.
To find a workaround, could you tell me how impactful this problem is for you? Does sagan work well after the FD is reopened?

Best regards,
Attila


________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Charles Eidsness <charles at ccxtechnologies.com>
Sent: Friday, June 21, 2019 6:22 PM
To: syslog-ng at lists.balabit.hu
Subject: [syslog-ng] Periodic "Destination timed out, reaping" and disconnection on a write fifo (pipe)

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

I kind of feel like this is going to be a dumb question but I just can't figure this out.

I'm trying to get syslog-ng working with sagan, but every 60 seconds syslog-ng closes the fifo it's writing to, which closes the sagan connection to the fifo. When running in verbose mode syslog-ng is throwing up a "Destination timed out, reaping..." message.

I simplified my syslog-ng configuration to:

source s_sys {
    system();
};

destination d_fifo {
    pipe(
        "/tmp/test.fifo"
    );
};

log {
    source(s_sys);
    destination(d_fifo);
};

And then run use cat to read the fifo:
cat /tmp/test.fifo

It sends all of the log traffic every but every 60 seconds cat closes. Is this expected behavior? Am I missing something?

I'm using syslog-ng version 3.19.1.

Thanks!
Charles

==
Charles Eidsness
Chief Engineer
CCX Technologies
(613)791-9715
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190624/a4a94f45/attachment.html>