[syslog-ng] Asynchronous address resolution using getaddrinfo_a()

Thu Jul 26 09:05:51 UTC 2018

Hi Laci and Scheidler,

Thanks a lot of discussing it internally and giving me inputs.
I gave further thought for all the inputs provided and I have come up with
a patch which I have attached with this email. Appreciate if you folks can
give any feedback on the patch. Some points about the fix.

   1. I didn't have to take either of the thread/signal callback
   approaches, there was another way give in the man page example. The
   approach mentioned in the example can easily take advantage of the existing
   timer infrastructure. This would mean less architectural changes.
   2. Our application is restricted to Linux hence I didn't have to think
   about cross platform scenarios.
   3. After browsing through the code, I felt the need to retain the
   synchronous way for places where there is not timer mechanism to retry.
   Which means async approach will need more code changes. Hence I have a if
   condition checking to see what is the preferred method of resolution the
   caller expects.
   4. The ivykis dns resolution might be a much cleaner approach. As you
   rightly mentioned, we are not sure of the timeline and effort to take in
   that change.
   5. A question about dns service that has bothered (I had also evauated
   using a local dns cache like bind) is what happens when TTL expires. The
   service would go look up from the dns server on next access right? What if
   the DNS link goes down during such a look up after TTL expiry. Won't we end
   up in the same problem that we have now?

Thanks,
Naveen

On Tue, Jul 24, 2018 at 2:30 AM Scheidler, Balázs <
balazs.scheidler at oneidentity.com> wrote:

> Hi,
>
> Another issue is what we would do with the incoming message flow until we
> are waiting for the result of the resolution.
>
> On Tue, Jul 24, 2018 at 8:56 AM, Szemere, László <
> laszlo.szemere at balabit.com> wrote:
>
>> Hello Naveen,
>>  thank you very much for the investigation and giving a proposal to the
>> issue. I successfully reproduced the blocking behavior with the faulty DNS
>> server in resolv.conf.
>>  Before answering you, we discussed the topic internally, and I would
>> like to add a few notes to your email. (And maybe start a common discussion
>> about the topic.)
>>
>>  1)
>>  The connection handling in afsocket is running in the main thread (This
>> is where the blocking behavior comes.), and mainly this is the reason why
>> it has many callback functions. Unfortunately it makes the code more
>> complex, and harder to maintain/debug.
>>  You clearly recognized a good point in the code
>> (afsocket_dd_try_connect) to introduce some asynchronous solution for DNS
>> resolving. I think there is no need to handle the "first attempt" any
>> different than the others.
>>
>>  2) Signal vs thread
>>  You have absolutely control over this, there are examples in our code
>> for both of them.
>>  IMHO with signals there might be a chance to conflict with other
>> components. (We recently had an issue with Java, but nothing which can not
>> be sorted out.) This is clearly just a personal opinion.
>>
>>  note: See examples of using "main_loop_call". If you only use those
>> callbacks to add some tasks to the main loop, than you don't have to deal
>> with parallelism, and it will make the rest of the code independent from
>> the chosen callback method.
>>
>>  3)
>>  Couple of months ago there were a discussion about adding async DNS
>> features to the alredy used ivykis library:
>> https://sourceforge.net/p/libivykis/mailman/message/36311243/
>>  Unfortunately there is no action in the topic since that, so I do not
>> recommend for you to wait for the final implementation. You can easily
>> start your own, or contribute to ivykis.
>>  However I think the idea is good: Introduce DNS resolving as an internal
>> module or service. At least one should keep in mind during the refactor of
>> afsocket, to make the DNS "service" interchangeable. (If we could gather
>> TTL information beside the resolving, it can be completely independent from
>> the main thread.)
>>
>>  4)
>>  The getaddrinfo_a is a GNU extension, so it might not be available on
>> all syslog-ng supported platforms. (There is already a branching in
>> https://github.com/balabit/syslog-ng/blob/e0ecad3dfafe5f34f7a5d2893b6a518e85ce3753/lib/host-resolve.c#L205
>> , so this is just a note, to not forget it.)
>>
>>
>> Best regards,
>> Laci
>>
>>
>> On Sat, Jul 21, 2018 at 3:39 AM, Naveen Revanna <raveenr at gmail.com>
>> wrote:
>>
>>> Hi Developers,
>>>
>>> When DNS server is unreachable, getaddrinfo() function will block (until
>>> it times out after few tens of seconds). If syslog-ng application is
>>> configured with a remote syslog server using its hostname,
>>> afsocket_dd_try_connect() will try to resolve this address in a loop (using
>>> a timer). Since getaddrinfo() is blocking, execution of this task will take
>>> few seconds thereby delaying the execution of other tasks. Eventually this
>>> will reach a state in which there will be lot of backlogs of tasks and
>>> syslog() appears to hang, delaying the execution of all shell commands (by
>>> upto a minute).
>>>
>>> Resolving the remote syslog server's address by having an entry in
>>> /etc/host could be a possible workaround (as indicated in a previous
>>> thread). However, this is not a desirable solution for our use case.
>>>
>>> Here is my thought on a possible fix that I am thinking to work on for
>>> which I am looking for feedback.
>>>
>>>    1. I am thinking of using getaddrinfo_a() (
>>>    http://man7.org/linux/man-pages/man3/getaddrinfo_a.3.html) in the
>>>    afsocket_dd_try_connect() loop function. We can have this async call only
>>>    in case it starts looping and not for the first attempt. The first attempt
>>>    can continue to do the current sync way.
>>>    2. If I were to take the above approach, there are two ways in which
>>>    we can know the status of address resolution. Which one do you folks think
>>>    would be better:
>>>       1. SIGEV_SIGNAL: We can receive a signal when a look-up
>>>       completes. We can take further action in the handler function.
>>>       2. SIGEV_THREAD: A notification function will be called. This
>>>       results in creation of a new thread (pthread?). What I am not sure of is
>>>       the impact of this thread creation on the existing thread infrastructure in
>>>       syslog-ng through ivykis.
>>>
>>>
>>> Repro:
>>> It is fairly easy to reproduce this issue
>>>
>>>    1. Configure a remote syslog server using it's hostname.
>>>    2. Make the dns unreachable (Edit resolve.conf and put wrong IP(s)
>>>    for 'nameserver' entries)
>>>    3. $ syslog-ng-ctl reload
>>>    4. Observe that any command executed on the shell takes unusually
>>>    long time.
>>>
>>> I can elaborate on any of the items here if something is not clear.
>>> Appreciate any pointers.
>>>
>>> Thanks,
>>> Naveen
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

-- 
--Naveen R
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180726/7ce64499/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: syslog-ng-hanging-fix.patch
Type: application/octet-stream
Size: 7468 bytes
Desc: not available
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180726/7ce64499/attachment-0001.obj>