<div dir="ltr"><div>Hi,<br><br></div>I have pushed the fix on the hotfix/3.4/full-disk-crash-fix branch on the github repository. Hopefully Algernon picks it up soon enough to master, but I&#39;m not sure when he plans a release in 3.4<br>
<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jun 15, 2013 at 7:06 PM, Balazs Scheidler <span dir="ltr">&lt;<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Thanks for the detailed report. Seems to be a use after free or double free in the error path. I&#39;ll have a closer look.</p>

<div class="gmail_quote"><div><div class="h5">On Jun 15, 2013 2:15 AM, &quot;Johnson, Chris (HP TippingPoint Roseville)&quot; &lt;<a href="mailto:chris.johnson3@hp.com" target="_blank">chris.johnson3@hp.com</a>&gt; wrote:<br type="attribution">
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">






<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal">I&#39;ve come across a situation where syslog-ng (3.3.3 and 3.3.9) aborts after trying to write to a disk that has no space and then is cleaned up.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I&#39;ve been able to reproduce the &#39;error&#39; condition with the following Set up:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">In the config file:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">###########################################################<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;"># test log destination<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">#<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">filter f_test{program(&quot;LOGID_99-*&quot; type(&quot;glob&quot;));};<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">destination d_test { file(&quot;/var/testpartition/test.log&quot; perm(0644) flags(no-multi-line)); };<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">log {<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">        source(s_local);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">        filter(f_test);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">        destination(d_test);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">        flags(final);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:&quot;Courier New&quot;">};<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p><u></u><span>1)<span style="font:7.0pt &quot;Times New Roman&quot;">     
</span></span><u></u>Fill up the &#39;/var/testpartition&#39; disk:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">root@device:/var/testpartition# dd if=/dev/zero of=/var/testpartition/foo bs=1M<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">dd: writing &#39;/var/testpartition/foo&#39;: No space left on device<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">3531+0 records in<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">3529+1 records out<u></u><u></u></span></p>
<p><u></u><span>2)<span style="font:7.0pt &quot;Times New Roman&quot;">     
</span></span><u></u>Send messages (<i><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">&#39;TEST: pre I/O error&#39;</span></i><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">)</span>to &#39;test.log&#39; until syslog-ng complains:<u></u><u></u></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14 22:41:09.623 [device] [syslog-ng-ERROR:] &quot;I/O error occurred while writing; fd=&#39;22&#39;, error=&#39;No space left on device (28)&#39;&quot;<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14 22:41:09.623 [device] [syslog-ng-NOTICE:] &quot;Suspending write operation because of an I/O error; fd=&#39;22&#39;, time_reopen=&#39;60&#39;&quot;<u></u><u></u></span></p>


<p><u></u><span>3)<span style="font:7.0pt &quot;Times New Roman&quot;">     
</span></span><u></u>Restart syslog-ng with a SIGHUP<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">root@device:/var/testpartition# kill -s HUP $(cat /var/run/syslog-ng.pid)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">Program received signal SIGHUP, Hangup.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">0x00007f38ee8dad63 in __epoll_wait_nocancel () from /xxx/xxx/xxx/xxx/xxx/xxx/lib/libc.so.6<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">(gdb) continue<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">Continuing.<u></u><u></u></span></p>
<p><u></u><span>4)<span style="font:7.0pt &quot;Times New Roman&quot;">     
</span></span><u></u>Send another message (<i><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">&#39;TEST: post I/O error; post SIGHUP&#39;</span></i><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">)</span>to syslog-ng (it complains):<u></u><u></u></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14 22:41:37.147 [device] [syslog-ng-NOTICE:] &quot;Configuration reload request received, reloading configuration;&quot;<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14 22:41:59.610 [device] [syslog-ng-ERROR:] &quot;I/O error occurred while writing; fd=&#39;23&#39;, error=&#39;No space left on device (28)&#39;&quot;<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14 22:41:59.610 [device] [syslog-ng-NOTICE:] &quot;Suspending write operation because of an I/O error; fd=&#39;23&#39;, time_reopen=&#39;60&#39;&quot;<u></u><u></u></span></p>


<p><u></u><span>5)<span style="font:7.0pt &quot;Times New Roman&quot;">     
</span></span><u></u>Clean up &#39;/var/testpartition&#39; disk:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">root@device:/var/testpartition# rm foo<u></u><u></u></span></p>
<p><u></u><span>6)<span style="font:7.0pt &quot;Times New Roman&quot;">     
</span></span><u></u>Send another message (<i><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">&#39;TEST: post I/O error; post SIGHUP 2&#39;</span></i><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">)</span>; syslog-ng aborts:<u></u><u></u></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">Program received signal SIGABRT, Aborted.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">0x00007f38ee83eda5 in raise (sig=&lt;value optimized out&gt;) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">        in ../nptl/sysdeps/unix/sysv/linux/raise.c<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">(gdb) bt<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#0  0x00007f38ee83eda5 in raise (sig=&lt;value optimized out&gt;) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#1  0x00007f38ee8402c3 in abort () at abort.c:88<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#2  0x00007f38ee87af18 in __libc_message (do_abort=2, fmt=0x7f38ee927cc8 &quot;*** glibc detected *** %s: %s: 0x%s ***\n&quot;)<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">    at ../sysdeps/unix/sysv/linux/libc_fatal.c:170<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#3  0x00007f38ee8804b8 in malloc_printerr (action=2, str=0x7f38ee927dd0 &quot;double free or corruption (!prev)&quot;,<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">    ptr=&lt;value optimized out&gt;) at malloc.c:5891<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#4  0x00007f38ee8825c6 in __libc_free (mem=0x7f38ee91e820) at malloc.c:3626<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#5  0x00007f38ef7d989a in ?? ()<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#6  0x0000000000401800 in ?? () at elf-init.c:99<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#7  0x00007f38efcf0257 in log_proto_file_writer_flush (s=0x674de0) at logproto.c:306<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#8  0x00007f38efcf04a8 in log_proto_file_writer_post (s=0x674de0,<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">    msg=0x6558a0 &quot;2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n&quot;, msg_len=89,<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">    consumed=0x7fffa53e570c) at logproto.c:380<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#9  0x00007f38efcfc274 in log_proto_post (s=0x674de0,<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">    msg=0x6558a0 &quot;2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n&quot;, msg_len=89,<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">    consumed=0x7fffa53e570c) at logproto.h:95<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#10 0x00007f38efcfc095 in log_writer_flush (self=0x655a30, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:983<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#11 0x00007f38efcf9be3 in log_writer_work_perform (s=0x655a30) at logwriter.c:129<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#12 0x00007f38efcf9efe in log_writer_io_flush_output (s=0x655a30) at logwriter.c:198<u></u><u></u></span></p>


<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#13 0x00007f38efd238b5 in iv_run_tasks (st=0x6036a0) at iv_task.c:46<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#14 0x00007f38efd22a86 in iv_main () at iv_main.c:266<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#15 0x00007f38efcfedf9 in main_loop_run () at mainloop.c:736<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">#16 0x00000000004017c9 in main (argc=1, argv=0x7fffa53e5968) at main.c:263<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The last two test messages <b>are</b> being written (run together with the last partial message that generated the &#39;disk full&#39; I/O error):<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">root@device:/var/testpartition# tail test.log<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:40:58.140+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:40:59.150+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:00.180+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:01.176+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:02.170+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:03.010+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:03.930+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:05.040+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:06.180+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">2013-06-14T22:41:07.140+00:00 device LOGID_99-INFO: T2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2013-06-14T22:42:30.476+00:00
 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:&quot;Courier New&quot;">root@device:/var/testpartition#<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Chris<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">----------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">Christopher Johnson<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas"><a href="mailto:chris.johnson3@hp.com" target="_blank">chris.johnson3@hp.com</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">HP Software - Security Product Group<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas"><a href="tel:%28916%29%20785-2817" value="+19167852817" target="_blank">(916) 785-2817</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">----------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>

<br></div></div>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>
</blockquote></div><br><br clear="all"><br>-- <br>Bazsi
</div>