<div dir="ltr"><div>Hi,<br><br></div>And here comes the fix:<br><a href="https://github.com/balabit/syslog-ng/pull/1218">https://github.com/balabit/syslog-ng/pull/1218</a><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Oct 1, 2016 at 5:10 PM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,<br><br></div>I could reproduce the issue in a localized environment. Thanks for your efforts, hopefully the fix is underway, once I can diagnose it properly.<br><br></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Fri, Sep 30, 2016 at 8:47 AM, Claus Albøge <span dir="ltr"><<a href="mailto:ca@tdchosting.dk" target="_blank">ca@tdchosting.dk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Just a quick update, after changing "dns-cache-size” to 10000, syslog-ng has been running for 3 days, without any restarts due to the SIGABRT.<br>
<br>
I guess the issues can be easily reproduced by setting “dns-cache-size” to 1.<br>
<br>
This issues is by the way related to the “Listen backlog issue” reported in: <a href="https://lists.balabit.hu/pipermail/syslog-ng/2016-September/023228.html" rel="noreferrer" target="_blank">https://lists.balabit.hu/piper<wbr>mail/syslog-ng/2016-September/<wbr>023228.html</a> (and <a href="https://github.com/balabit/syslog-ng/pull/1211" rel="noreferrer" target="_blank">https://github.com/balabit/sys<wbr>log-ng/pull/1211</a>). The sudden surge in SYN packets was caused because syslog-ng restarted due to the SIGABRT, causing all connected clients to reconnect. With a couple of thousand clients, it was easy to trigger a situation, where more than 255 clients reconnected at once, causing the hardcoded listen backlog to run full.<br>
<br>
<br>
<br>
/Claus A<br>
<div class="m_2513136098818546066HOEnZb"><div class="m_2513136098818546066h5"><br>
<br>
> On 26. sep. 2016, at 13.53, Claus Albøge <<a href="mailto:ca@tdchosting.dk" target="_blank">ca@tdchosting.dk</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I tried to minimize my config as much as possible, but was unable to reproduce - had it run for a couple of hours.<br>
><br>
> Looking at the source code pointed my towards “effective_dns_cache_options" and dns-cache-size (<a href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/examples-dns.html" rel="noreferrer" target="_blank">https://www.balabit.com/docum<wbr>ents/syslog-ng-ose-latest-<wbr>guides/en/syslog-ng-ose-guide-<wbr>admin/html/examples-dns.html</a>)<br>
><br>
> Setting dns-cache-size to 1 on my minimized config triggered the SIGABRT within a few seconds. (Default value seems to be 1007!, which is higher that the number of hosts sending logs to my syslog-ng instance with the minimized config)<br>
><br>
> Setting dns-cache-size to 10000 on a syslog-instance with the “full” number of log sources, seems to work around the issue, as syslog-ng has been running for almost a hour now without any restarts - normally it restarts every few minutes.<br>
><br>
> So it does indeed seems like an issue with the dnscache logic. You might be able to reproduce by setting dns-cache-size to a very low number.<br>
><br>
> If you still need a core dump to debug this further, I might need a bit of help creating the coredump correctly. After the SIGABRT, running gcore in gdb creates a core, but if I try to read this core again using gdb, I’m unable to do a backtrace. Do I need to do anything special?<br>
><br>
><br>
> /Claus A<br>
><br>
><br>
>> On 23. sep. 2016, at 08.37, Claus Albøge <<a href="mailto:ca@tdchosting.dk" target="_blank">ca@tdchosting.dk</a>> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> I’ll try to reproduce with a minimal configuration, without ssl-certs, etc.<br>
>><br>
>> Syslog-ng binaries are from: <a href="https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-7-x86_64/00449829-syslog-ng/" rel="noreferrer" target="_blank">https://copr-be.cloud.fedorapr<wbr>oject.org/results/czanik/<wbr>syslog-ng38/epel-7-x86_64/<wbr>00449829-syslog-ng/</a><br>
>><br>
>><br>
>> /Claus A<br>
>><br>
>>> On 23. sep. 2016, at 08.25, Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>> wrote:<br>
>>><br>
>>> Is there a possibility for me to receive the core file and the binaries? (Syslog-ng plus dependent libs)<br>
>>><br>
>>> This seems to be some kind of dns cache corruption, which is difficult to diagnose without those.<br>
>>><br>
>>> Bazsi<br>
>>><br>
>>><br>
>>> On Sep 22, 2016 3:29 PM, "Claus Albøge" <<a href="mailto:ca@tdchosting.dk" target="_blank">ca@tdchosting.dk</a>> wrote:<br>
>>> Hi,<br>
>>><br>
>>> # gdb --args /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid<br>
>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7<br>
>>> Copyright (C) 2013 Free Software Foundation, Inc.<br>
>>> License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" rel="noreferrer" target="_blank">http://gnu.org/licenses/gpl.h<wbr>tml</a>><br>
>>> This is free software: you are free to change and redistribute it.<br>
>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>
>>> and "show warranty" for details.<br>
>>> This GDB was configured as "x86_64-redhat-linux-gnu".<br>
>>> For bug reporting instructions, please see:<br>
>>> <<a href="http://www.gnu.org/software/gdb/bugs/" rel="noreferrer" target="_blank">http://www.gnu.org/software/g<wbr>db/bugs/</a>>...<br>
>>> Reading symbols from /usr/sbin/syslog-ng...Reading symbols from /usr/lib/debug/usr/sbin/syslog<wbr>-ng.debug...done.<br>
>>> done.<br>
>>> (gdb) set pagination off<br>
>>> (gdb) run<br>
>>> Starting program: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid<br>
>>> [Thread debugging using libthread_db enabled]<br>
>>> Using host libthread_db library "/lib64/libthread_db.so.1".<br>
>>> [New Thread 0x7ffff1e8a700 (LWP 12861)]<br>
>>> [New Thread 0x7ffff1689700 (LWP 12862)]<br>
>>> [New Thread 0x7ffff0e88700 (LWP 12863)]<br>
>>> [New Thread 0x7fffe3fff700 (LWP 13103)]<br>
>>> [New Thread 0x7fffe37fe700 (LWP 13108)]<br>
>>> [New Thread 0x7fffe2ffd700 (LWP 13109)]<br>
>>> [New Thread 0x7fffe27fc700 (LWP 13110)]<br>
>>> [New Thread 0x7fffe1ffb700 (LWP 13111)]<br>
>>> [Thread 0x7ffff1689700 (LWP 12862) exited]<br>
>>> [Thread 0x7ffff1e8a700 (LWP 12861) exited]<br>
>>> [New Thread 0x7ffff1e8a700 (LWP 13395)]<br>
>>> [New Thread 0x7ffff1689700 (LWP 13396)]<br>
>>> [Thread 0x7fffe37fe700 (LWP 13108) exited]<br>
>>> [Thread 0x7fffe2ffd700 (LWP 13109) exited]<br>
>>> [New Thread 0x7fffe37fe700 (LWP 13646)]<br>
>>> [New Thread 0x7fffe2ffd700 (LWP 13647)]<br>
>>> [Thread 0x7ffff1689700 (LWP 13396) exited]<br>
>>> [Thread 0x7fffe3fff700 (LWP 13103) exited]<br>
>>> [New Thread 0x7fffe3fff700 (LWP 13959)]<br>
>>> [New Thread 0x7ffff1689700 (LWP 13960)]<br>
>>> [Thread 0x7ffff1e8a700 (LWP 13395) exited]<br>
>>> [Thread 0x7fffe1ffb700 (LWP 13111) exited]<br>
>>> [New Thread 0x7fffe1ffb700 (LWP 14040)]<br>
>>> [New Thread 0x7ffff1e8a700 (LWP 14041)]<br>
>>> [Thread 0x7fffe3fff700 (LWP 13959) exited]<br>
>>> [Thread 0x7ffff0e88700 (LWP 12863) exited]<br>
>>> [New Thread 0x7ffff0e88700 (LWP 14182)]<br>
>>> [New Thread 0x7fffe3fff700 (LWP 14183)]<br>
>>> **<br>
>>> ERROR:lib/dnscache.c:112:dns_c<wbr>ache_key_hash: code should not be reached<br>
>>><br>
>>> Program received signal SIGABRT, Aborted.<br>
>>> [Switching to Thread 0x7ffff0e88700 (LWP 14182)]<br>
>>> 0x00007ffff5fb05f7 in raise () from /lib64/libc.so.6<br>
>>> Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.163-3.el7.x8<wbr>6_64 elfutils-libs-0.163-3.el7.x86_<wbr>64 eventlog-0.2.13-4.el7.x86_64 glib2-2.42.2-5.el7.x86_64 glibc-2.17-106.el7_2.8.x86_64 ivykis-0.36.2-2.el7.x86_64 json-c-0.11-4.el7_0.x86_64 keyutils-libs-1.5.8-3.el7.x86_<wbr>64 krb5-libs-1.13.2-12.el7_2.x86_<wbr>64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libgcrypt-1.5.3-12.el7_1.1.x86<wbr>_64 libgpg-error-1.12-3.el7.x86_64 libnet-1.1.6-7.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libuuid-2.23.2-26.el7_2.3.x86_<wbr>64 openssl-libs-1.0.1e-51.el7_2.5<wbr>.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-19.el7_2.13.x<wbr>86_64 tcp_wrappers-libs-7.6-77.el7.x<wbr>86_64 xz-libs-5.1.2-12alpha.el7.x86_<wbr>64 zlib-1.2.7-15.el7.x86_64<br>
>>> (gdb) bt full<br>
>>> #0 0x00007ffff5fb05f7 in raise () from /lib64/libc.so.6<br>
>>> No symbol table info available.<br>
>>> #1 0x00007ffff5fb1ce8 in abort () from /lib64/libc.so.6<br>
>>> No symbol table info available.<br>
>>> #2 0x00007ffff70426d5 in g_assertion_message () from /lib64/libglib-2.0.so.0<br>
>>> No symbol table info available.<br>
>>> #3 0x00007ffff704276a in g_assertion_message_expr () from /lib64/libglib-2.0.so.0<br>
>>> No symbol table info available.<br>
>>> #4 0x00007ffff7b62cc4 in dns_cache_key_hash (e=<optimized out>) at lib/dnscache.c:112<br>
>>> e = <optimized out><br>
>>> #5 0x00007ffff700c4fc in g_hash_table_remove_internal () from /lib64/libglib-2.0.so.0<br>
>>> No symbol table info available.<br>
>>> #6 0x00007ffff7b67584 in resolve_sockaddr_to_inet_or_in<wbr>et6_hostname (host_resolve_options=0x651754<wbr>, saddr=0x183d350, result_len=0x7ffff0e85370) at lib/host-resolve.c:332<br>
>>> hname = 0x7ffff0e864b8 "213.174.76.60"<br>
>>> hname_len = 5<br>
>>> positive = 0<br>
>>> dnscache_key = 0x183d368<br>
>>> #7 resolve_sockaddr_to_hostname (result_len=result_len@entry=0<wbr>x7ffff0e85370, saddr=0x183d350, host_resolve_options=0x651754) at lib/host-resolve.c:343<br>
>>> No locals.<br>
>>> #8 0x00007ffff7b6c03a in log_source_mangle_hostname (self=self@entry=0x1868c10, msg=msg@entry=0x7fffe87cdd40) at lib/logsource.c:156<br>
>>> resolved_name = <optimized out><br>
>>> resolved_name_len = 140737093885344<br>
>>> #9 0x00007ffff7b6c280 in log_source_queue (s=0x1868c10, msg=0x7fffe87cdd40, path_options=0x7ffff0e85550, user_data=<optimized out>) at lib/logsource.c:266<br>
>>> self = 0x1868c10<br>
>>> i = <optimized out><br>
>>> __func__ = "log_source_queue"<br>
>>> #10 0x00007ffff7b6ca0a in log_pipe_queue (path_options=<optimized out>, msg=<optimized out>, s=<optimized out>) at lib/logpipe.h:339<br>
>>> No locals.<br>
>>> #11 log_pipe_forward_msg (self=<optimized out>, path_options=<optimized out>, msg=<optimized out>) at lib/logpipe.h:302<br>
>>> No locals.<br>
>>> #12 log_pipe_queue (path_options=<optimized out>, msg=<optimized out>, s=<optimized out>) at lib/logpipe.h:343<br>
>>> No locals.<br>
>>> #13 log_pipe_forward_msg (self=<optimized out>, path_options=<optimized out>, msg=<optimized out>) at lib/logpipe.h:302<br>
>>> No locals.<br>
>>> #14 log_pipe_queue (path_options=<optimized out>, msg=0x7fffe87cdd40, s=0x7fffe87cdd40) at lib/logpipe.h:343<br>
>>> No locals.<br>
>>> #15 log_source_post (self=self@entry=0x1868c10, msg=msg@entry=0x7fffe87cdd40) at lib/logsource.c:249<br>
>>> path_options = {ack_needed = 1, flow_control_requested = 0, matched = 0x0}<br>
>>> old_window_size = <optimized out><br>
>>> __func__ = "log_source_post"<br>
>>> #16 0x00007ffff7b6b179 in log_reader_handle_line (aux=0x7ffff0e85600, length=<optimized out>, line=<optimized out>, self=0x1868c10) at lib/logreader.c:320<br>
>>> m = 0x7fffe87cdd40<br>
>>> #17 log_reader_fetch_log (self=self@entry=0x1868c10) at lib/logreader.c:381<br>
>>> bookmark = <optimized out><br>
>>> msg = 0x7fffe445cef5 "<14>1 2016-09-22T10:51:07.178337+02:<wbr>00 <a href="http://pafodev02.int.patientforsikringen.dk" rel="noreferrer" target="_blank">pafodev02.int.patientforsikrin<wbr>gen.dk</a> Microsoft-Windows-Security-Aud<wbr>iting 608 - [NXLOG@47121 AssetID=\"105939\" CustomerID=\"0002592\" Location=\"Hyper-V - Farm 01 (ps"...<br>
>>> msg_len = 4245<br>
>>> status = <optimized out><br>
>>> msg_count = 1<br>
>>> may_read = 1<br>
>>> aux = {peer_addr = 0x0, data = "\000\207\350\360\377\177\000\<wbr>000)\000\000\000\000\000\000\0<wbr>00\304\335U\366\377\177\000\00<wbr>0\000Y\350\360\377\177\000\000<wbr>\230\235\332\001\000\000\000\<wbr>000@ \245\350\377\177\000\000h\313U<wbr>\366\377\177\000\000@V\350\<wbr>360\377\177\000\000@V\350\360\<wbr>377\177\000\000\000Y\350\360\<wbr>377\177\000\000\340\037\245\<wbr>350\377\177\000\000\b\000\000\<wbr>000\000\000\000\000\250\037\<wbr>245\350\377\177\000\000\200V\<wbr>350\360\377\177\000\000\223\<wbr>341U\366\377\177\000\000\001\<wbr>000\000\000\000\000\000\000\<wbr>006Y\350\360\377\177\000\000\<wbr>aY\350\360\377\177\000\000\<wbr>000Y\350\360\377\177\000\000\<wbr>aY\350\360\377\177\000\000gk\<wbr>266\367\377\177\000\000\220", '\000' <repeats 15 times>, "\240 E\001\000\000\000\000\301\326\<wbr>003\367\377\177\000\000"..., end_ptr = 0}<br>
>>> __func__ = "log_reader_fetch_log"<br>
>>> #18 0x00007ffff7b6b299 in log_reader_work_perform (s=0x1868c10) at lib/logreader.c:88<br>
>>> self = 0x1868c10<br>
>>> #19 0x00007ffff7b7307d in _work (self=<optimized out>) at lib/mainloop-io-worker.c:52<br>
>>> No locals.<br>
>>> #20 0x00007ffff655dc67 in iv_work_thread_do_work () from /lib64/libivykis.so.0<br>
>>> No symbol table info available.<br>
>>> #21 0x00007ffff655cfc3 in iv_run_tasks () from /lib64/libivykis.so.0<br>
>>> No symbol table info available.<br>
>>> #22 0x00007ffff655f7dc in iv_main () from /lib64/libivykis.so.0<br>
>>> No symbol table info available.<br>
>>> #23 0x00007ffff655dab4 in iv_work_thread () from /lib64/libivykis.so.0<br>
>>> No symbol table info available.<br>
>>> #24 0x00007ffff6560270 in iv_thread_handler () from /lib64/libivykis.so.0<br>
>>> No symbol table info available.<br>
>>> #25 0x00007ffff6344dc5 in start_thread () from /lib64/libpthread.so.0<br>
>>> No symbol table info available.<br>
>>> #26 0x00007ffff6071ced in clone () from /lib64/libc.so.6<br>
>>> No symbol table info available.<br>
>>> (gdb) frame 6<br>
>>> #6 0x00007ffff7b67584 in resolve_sockaddr_to_inet_or_in<wbr>et6_hostname (host_resolve_options=0x651754<wbr>, saddr=0x183d350, result_len=0x7ffff0e85370) at lib/host-resolve.c:332<br>
>>> 332 dns_caching_store(saddr->sa.s<wbr>a_family, dnscache_key, hname, positive);<br>
>>> (gdb) print *saddr<br>
>>> $1 = {refcnt = {counter = 3}, flags = 0, sa_funcs = 0x7ffff7dd7060 <inet_sockaddr_funcs>, salen = 16, sa = {sa_family = 2, sa_data = "\225\327ծL<\000\000\000\000\0<wbr>00\000\000"}}<br>
>>> (gdb) print *saddr->sa<br>
>>> Structure has no component named operator*.<br>
>>> (gdb) print saddr<br>
>>> $2 = (GSockAddr *) 0x183d350<br>
>>><br>
>>><br>
>>> /Claus A<br>
>>><br>
>>><br>
>>>> On 21. sep. 2016, at 15.37, Szalai, Attila <<a href="mailto:Attila.Szalai@morganstanley.com" target="_blank">Attila.Szalai@morganstanley.c<wbr>om</a>> wrote:<br>
>>>><br>
>>>> Hi,<br>
>>>><br>
>>>> If it is convenient to you, just for the sake of my curiosity, can you run some more command in the gdb session, after you reproduced the issue?<br>
>>>><br>
>>>> * bt full<br>
>>>> * frame 6 (you should search for the line with resolve_sockaddr_to_inet_or_in<wbr>et6_hostname and write the number written in that line. In the case of your previous email, it was #6)<br>
>>>> * print *saddr<br>
>>>> * print *saddr->sa<br>
>>>><br>
>>>><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>______________________________<wbr>__________________<br>
>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
>> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
>> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
>><br>
><br>
><br>
> ______________________________<wbr>______________________________<wbr>__________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
><br>
<br>
<br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br><div class="m_2513136098818546066gmail_signature" data-smartmail="gmail_signature">Bazsi</div>
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Bazsi</div>
</div>