[syslog-ng] Asynchronous address resolution using getaddrinfo_a()

[Naveen Revanna]

Hi Developers,

When DNS server is unreachable, getaddrinfo() function will block (until it
times out after few tens of seconds). If syslog-ng application is
configured with a remote syslog server using its hostname,
afsocket_dd_try_connect() will try to resolve this address in a loop (using
a timer). Since getaddrinfo() is blocking, execution of this task will take
few seconds thereby delaying the execution of other tasks. Eventually this
will reach a state in which there will be lot of backlogs of tasks and
syslog() appears to hang, delaying the execution of all shell commands (by
upto a minute).

Resolving the remote syslog server's address by having an entry in
/etc/host could be a possible workaround (as indicated in a previous
thread). However, this is not a desirable solution for our use case.

Here is my thought on a possible fix that I am thinking to work on for
which I am looking for feedback.

   1. I am thinking of using getaddrinfo_a() (
   http://man7.org/linux/man-pages/man3/getaddrinfo_a.3.html) in the
   afsocket_dd_try_connect() loop function. We can have this async call only
   in case it starts looping and not for the first attempt. The first attempt
   can continue to do the current sync way.
   2. If I were to take the above approach, there are two ways in which we
   can know the status of address resolution. Which one do you folks think
   would be better:
      1. SIGEV_SIGNAL: We can receive a signal when a look-up completes. We
      can take further action in the handler function.
      2. SIGEV_THREAD: A notification function will be called. This results
      in creation of a new thread (pthread?). What I am not sure of is
the impact
      of this thread creation on the existing thread infrastructure in
syslog-ng
      through ivykis.


Repro:
It is fairly easy to reproduce this issue

   1. Configure a remote syslog server using it's hostname.
   2. Make the dns unreachable (Edit resolve.conf and put wrong IP(s) for
   'nameserver' entries)
   3. $ syslog-ng-ctl reload
   4. Observe that any command executed on the shell takes unusually long
   time.

I can elaborate on any of the items here if something is not clear.
Appreciate any pointers.

Thanks,
Naveen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180720/5cf0bf8a/attachment.html>