[syslog-ng] syslog-ng 3.19 not processing /dev/log on RH#L6

[Evan Rempel]

We did not see this message on the local host, nor on either of our syslog servers (the host should send all logs to both servers).

P.S. This is on RHEL6 so there is no journal.

Evan.

On 1/29/19 3:26 AM, Szakacs, Attila wrote:
> Hi Evan,
>
> After we fork the process, if the initialization failed in the new deamon, first we wait for it to exit, then if it did not exit in 3 seconds, we send SIGTERM, then SIGKILL to it.
> If even those signals could not not terminate the process, we let it be, and start a new. In this case we log the following:
> "Initialization failed but the daemon did not exit, even when forced to, trying to recover"
>
> Could you please check, whether you have seen this log somewhere? (journal, etc...)
>
> Best regards,
> Attila
>
> On Mon, Jan 28, 2019 at 10:11 AM Szakacs, Attila <attila.szakacs at balabit.com <mailto:attila.szakacs at balabit.com>> wrote:
>
>     Hi Evan,
>
>     Thank you for the heads-up!
>
>     This issue might be related to this PR: https://github.com/balabit/syslog-ng/pull/2099
>     I will try to reproduce it.
>
>     Best regards,
>     Attila
>
>     On Mon, Jan 21, 2019 at 9:51 PM Evan Rempel <erempel at uvic.ca <mailto:erempel at uvic.ca>> wrote:
>
>         We just had a network even in our environment that resulted in many hosts being isolated on the network in some very odd ways.
>         The hosts had some connections inbound, but no outbound and the network seemed to come and go very quickly.
>
>
>         syslog-ng detected that it lost connection to its down-stream syslog collectors a did the normal connection reaping. We have this set for 5 seconds.
>         At some point the syslog-ng process terminated and the supervisor process launched a new child. This worked well, but in
>         this odd senario this happened quickly and at on point the supervisor process failed to reap one of its defunct children but still launched a new child
>         process. The defunct process seemed to hold onto the /dev/log handle so the new child could not get access to it and the result is that
>         we lost hours of logs that flow through the /dev/log socket.
>         When we discovered the issue, there where two children of the supervisor process. One sas in a "defunct" state while the other was running,
>         but only logging kernel messages (from /dev/kmsg) and from the internal syslog-ng source.
>         I suspect that the child reaping process on SIGCHLD does not use a while loop and misses some children.
>
>         Evan.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190129/a5af21e5/attachment.html>