[syslog-ng] syslog-ng 3.19 not processing /dev/log on RH#L6

[Szakacs, Attila]

Hi Evan,

After we fork the process, if the initialization failed in the new deamon,
first we wait for it to exit, then if it did not exit in 3 seconds, we send
SIGTERM, then SIGKILL to it.
If even those signals could not not terminate the process, we let it be,
and start a new. In this case we log the following:
"Initialization failed but the daemon did not exit, even when forced to,
trying to recover"

Could you please check, whether you have seen this log somewhere? (journal,
etc...)

Best regards,
Attila

On Mon, Jan 28, 2019 at 10:11 AM Szakacs, Attila <attila.szakacs at balabit.com>
wrote:

> Hi Evan,
>
> Thank you for the heads-up!
>
> This issue might be related to this PR:
> https://github.com/balabit/syslog-ng/pull/2099
> I will try to reproduce it.
>
> Best regards,
> Attila
>
> On Mon, Jan 21, 2019 at 9:51 PM Evan Rempel <erempel at uvic.ca> wrote:
>
>> We just had a network even in our environment that resulted in many hosts
>> being isolated on the network in some very odd ways.
>> The hosts had some connections inbound, but no outbound and the network
>> seemed to come and go very quickly.
>>
>>
>> syslog-ng detected that it lost connection to its down-stream syslog
>> collectors a did the normal connection reaping. We have this set for 5
>> seconds.
>> At some point the syslog-ng process terminated and the supervisor process
>> launched a new child. This worked well, but in
>> this odd senario this happened quickly and at on point the supervisor
>> process failed to reap one of its defunct children but still launched a new
>> child
>> process. The defunct process seemed to hold onto the /dev/log handle so
>> the new child could not get access to it and the result is that
>> we lost hours of logs that flow through the /dev/log socket.
>> When we discovered the issue, there where two children of the supervisor
>> process. One sas in a "defunct" state while the other was running,
>> but only logging kernel messages (from /dev/kmsg) and from the internal
>> syslog-ng source.
>> I suspect that the child reaping process on SIGCHLD does not use a while
>> loop and misses some children.
>>
>> Evan.
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190129/8d88e288/attachment.html>