<p dir="ltr">Thanks for the detailed report. Seems to be a use after free or double free in the error path. I'll have a closer look.</p>
<div class="gmail_quote">On Jun 15, 2013 2:15 AM, "Johnson, Chris (HP TippingPoint Roseville)" <<a href="mailto:chris.johnson3@hp.com">chris.johnson3@hp.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal">I've come across a situation where syslog-ng (3.3.3 and 3.3.9) aborts after trying to write to a disk that has no space and then is cleaned up.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I've been able to reproduce the 'error' condition with the following Set up:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">In the config file:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">###########################################################<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""># test log destination<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">#<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">filter f_test{program("LOGID_99-*" type("glob"));};<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">destination d_test { file("/var/testpartition/test.log" perm(0644) flags(no-multi-line)); };<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">log {<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> source(s_local);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> filter(f_test);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> destination(d_test);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> flags(final);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">};<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p><u></u><span>1)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Fill up the '/var/testpartition' disk:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# dd if=/dev/zero of=/var/testpartition/foo bs=1M<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">dd: writing '/var/testpartition/foo': No space left on device<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">3531+0 records in<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">3529+1 records out<u></u><u></u></span></p>
<p><u></u><span>2)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Send messages (<i><span style="font-size:10.0pt;font-family:"Courier New"">'TEST: pre I/O error'</span></i><span style="font-size:10.0pt;font-family:"Courier New"">)</span>to 'test.log' until syslog-ng complains:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:09.623 [device] [syslog-ng-ERROR:] "I/O error occurred while writing; fd='22', error='No space left on device (28)'"<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:09.623 [device] [syslog-ng-NOTICE:] "Suspending write operation because of an I/O error; fd='22', time_reopen='60'"<u></u><u></u></span></p>
<p><u></u><span>3)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Restart syslog-ng with a SIGHUP<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# kill -s HUP $(cat /var/run/syslog-ng.pid)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Program received signal SIGHUP, Hangup.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">0x00007f38ee8dad63 in __epoll_wait_nocancel () from /xxx/xxx/xxx/xxx/xxx/xxx/lib/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">(gdb) continue<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Continuing.<u></u><u></u></span></p>
<p><u></u><span>4)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Send another message (<i><span style="font-size:10.0pt;font-family:"Courier New"">'TEST: post I/O error; post SIGHUP'</span></i><span style="font-size:10.0pt;font-family:"Courier New"">)</span>to syslog-ng (it complains):<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:37.147 [device] [syslog-ng-NOTICE:] "Configuration reload request received, reloading configuration;"<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:59.610 [device] [syslog-ng-ERROR:] "I/O error occurred while writing; fd='23', error='No space left on device (28)'"<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:59.610 [device] [syslog-ng-NOTICE:] "Suspending write operation because of an I/O error; fd='23', time_reopen='60'"<u></u><u></u></span></p>
<p><u></u><span>5)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Clean up '/var/testpartition' disk:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# rm foo<u></u><u></u></span></p>
<p><u></u><span>6)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Send another message (<i><span style="font-size:10.0pt;font-family:"Courier New"">'TEST: post I/O error; post SIGHUP 2'</span></i><span style="font-size:10.0pt;font-family:"Courier New"">)</span>; syslog-ng aborts:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Program received signal SIGABRT, Aborted.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">0x00007f38ee83eda5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> in ../nptl/sysdeps/unix/sysv/linux/raise.c<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">(gdb) bt<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#0 0x00007f38ee83eda5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#1 0x00007f38ee8402c3 in abort () at abort.c:88<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#2 0x00007f38ee87af18 in __libc_message (do_abort=2, fmt=0x7f38ee927cc8 "*** glibc detected *** %s: %s: 0x%s ***\n")<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> at ../sysdeps/unix/sysv/linux/libc_fatal.c:170<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#3 0x00007f38ee8804b8 in malloc_printerr (action=2, str=0x7f38ee927dd0 "double free or corruption (!prev)",<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> ptr=<value optimized out>) at malloc.c:5891<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#4 0x00007f38ee8825c6 in __libc_free (mem=0x7f38ee91e820) at malloc.c:3626<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#5 0x00007f38ef7d989a in ?? ()<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#6 0x0000000000401800 in ?? () at elf-init.c:99<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#7 0x00007f38efcf0257 in log_proto_file_writer_flush (s=0x674de0) at logproto.c:306<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#8 0x00007f38efcf04a8 in log_proto_file_writer_post (s=0x674de0,<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> msg=0x6558a0 "2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n", msg_len=89,<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> consumed=0x7fffa53e570c) at logproto.c:380<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#9 0x00007f38efcfc274 in log_proto_post (s=0x674de0,<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> msg=0x6558a0 "2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n", msg_len=89,<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> consumed=0x7fffa53e570c) at logproto.h:95<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#10 0x00007f38efcfc095 in log_writer_flush (self=0x655a30, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:983<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#11 0x00007f38efcf9be3 in log_writer_work_perform (s=0x655a30) at logwriter.c:129<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#12 0x00007f38efcf9efe in log_writer_io_flush_output (s=0x655a30) at logwriter.c:198<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#13 0x00007f38efd238b5 in iv_run_tasks (st=0x6036a0) at iv_task.c:46<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#14 0x00007f38efd22a86 in iv_main () at iv_main.c:266<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#15 0x00007f38efcfedf9 in main_loop_run () at mainloop.c:736<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#16 0x00000000004017c9 in main (argc=1, argv=0x7fffa53e5968) at main.c:263<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The last two test messages <b>are</b> being written (run together with the last partial message that generated the 'disk full' I/O error):<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# tail test.log<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:40:58.140+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:40:59.150+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:00.180+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:01.176+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:02.170+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:03.010+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:03.930+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:05.040+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:06.180+00:00 device LOGID_99-INFO: TEST: pre I/O error<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:07.140+00:00 device LOGID_99-INFO: T2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2013-06-14T22:42:30.476+00:00
device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition#<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Chris<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">----------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">Christopher Johnson<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas"><a href="mailto:chris.johnson3@hp.com" target="_blank">chris.johnson3@hp.com</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">HP Software - Security Product Group<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas"><a href="tel:%28916%29%20785-2817" value="+19167852817" target="_blank">(916) 785-2817</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">----------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>