<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1541358413;
mso-list-type:hybrid;
mso-list-template-ids:-1743463496 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal">I've come across a situation where syslog-ng (3.3.3 and 3.3.9) aborts after trying to write to a disk that has no space and then is cleaned up.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I've been able to reproduce the 'error' condition with the following Set up:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">In the config file:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">###########################################################<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""># test log destination<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">#<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">filter f_test{program("LOGID_99-*" type("glob"));};<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">destination d_test { file("/var/testpartition/test.log" perm(0644) flags(no-multi-line)); };<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">log {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> source(s_local);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> filter(f_test);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> destination(d_test);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> flags(final);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">};<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Fill up the '/var/testpartition' disk:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# dd if=/dev/zero of=/var/testpartition/foo bs=1M<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">dd: writing '/var/testpartition/foo': No space left on device<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">3531+0 records in<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">3529+1 records out<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Send messages (<i><span style="font-size:10.0pt;font-family:"Courier New"">'TEST: pre I/O error'</span></i><span style="font-size:10.0pt;font-family:"Courier New"">)</span>to 'test.log' until syslog-ng complains:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:09.623 [device] [syslog-ng-ERROR:] "I/O error occurred while writing; fd='22', error='No space left on device (28)'"<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:09.623 [device] [syslog-ng-NOTICE:] "Suspending write operation because of an I/O error; fd='22', time_reopen='60'"<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">3)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Restart syslog-ng with a SIGHUP<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# kill -s HUP $(cat /var/run/syslog-ng.pid)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Program received signal SIGHUP, Hangup.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">0x00007f38ee8dad63 in __epoll_wait_nocancel () from /xxx/xxx/xxx/xxx/xxx/xxx/lib/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">(gdb) continue<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Continuing.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">4)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Send another message (<i><span style="font-size:10.0pt;font-family:"Courier New"">'TEST: post I/O error; post SIGHUP'</span></i><span style="font-size:10.0pt;font-family:"Courier New"">)</span>to syslog-ng (it complains):<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:37.147 [device] [syslog-ng-NOTICE:] "Configuration reload request received, reloading configuration;"<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:59.610 [device] [syslog-ng-ERROR:] "I/O error occurred while writing; fd='23', error='No space left on device (28)'"<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14 22:41:59.610 [device] [syslog-ng-NOTICE:] "Suspending write operation because of an I/O error; fd='23', time_reopen='60'"<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">5)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Clean up '/var/testpartition' disk:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# rm foo<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">6)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Send another message (<i><span style="font-size:10.0pt;font-family:"Courier New"">'TEST: post I/O error; post SIGHUP 2'</span></i><span style="font-size:10.0pt;font-family:"Courier New"">)</span>; syslog-ng aborts:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Program received signal SIGABRT, Aborted.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">0x00007f38ee83eda5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> in ../nptl/sysdeps/unix/sysv/linux/raise.c<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">(gdb) bt<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#0 0x00007f38ee83eda5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#1 0x00007f38ee8402c3 in abort () at abort.c:88<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#2 0x00007f38ee87af18 in __libc_message (do_abort=2, fmt=0x7f38ee927cc8 "*** glibc detected *** %s: %s: 0x%s ***\n")<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> at ../sysdeps/unix/sysv/linux/libc_fatal.c:170<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#3 0x00007f38ee8804b8 in malloc_printerr (action=2, str=0x7f38ee927dd0 "double free or corruption (!prev)",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> ptr=<value optimized out>) at malloc.c:5891<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#4 0x00007f38ee8825c6 in __libc_free (mem=0x7f38ee91e820) at malloc.c:3626<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#5 0x00007f38ef7d989a in ?? ()<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#6 0x0000000000401800 in ?? () at elf-init.c:99<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#7 0x00007f38efcf0257 in log_proto_file_writer_flush (s=0x674de0) at logproto.c:306<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#8 0x00007f38efcf04a8 in log_proto_file_writer_post (s=0x674de0,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> msg=0x6558a0 "2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n", msg_len=89,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> consumed=0x7fffa53e570c) at logproto.c:380<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#9 0x00007f38efcfc274 in log_proto_post (s=0x674de0,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> msg=0x6558a0 "2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n", msg_len=89,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New""> consumed=0x7fffa53e570c) at logproto.h:95<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#10 0x00007f38efcfc095 in log_writer_flush (self=0x655a30, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:983<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#11 0x00007f38efcf9be3 in log_writer_work_perform (s=0x655a30) at logwriter.c:129<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#12 0x00007f38efcf9efe in log_writer_io_flush_output (s=0x655a30) at logwriter.c:198<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#13 0x00007f38efd238b5 in iv_run_tasks (st=0x6036a0) at iv_task.c:46<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#14 0x00007f38efd22a86 in iv_main () at iv_main.c:266<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#15 0x00007f38efcfedf9 in main_loop_run () at mainloop.c:736<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">#16 0x00000000004017c9 in main (argc=1, argv=0x7fffa53e5968) at main.c:263<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">(gdb)<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The last two test messages <b>are</b> being written (run together with the last partial message that generated the 'disk full' I/O error):<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition# tail test.log<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:40:58.140+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:40:59.150+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:00.180+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:01.176+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:02.170+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:03.010+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:03.930+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:05.040+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:06.180+00:00 device LOGID_99-INFO: TEST: pre I/O error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">2013-06-14T22:41:07.140+00:00 device LOGID_99-INFO: T2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2013-06-14T22:42:30.476+00:00
device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">root@device:/var/testpartition#<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Chris<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">----------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">Christopher Johnson<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas"><a href="mailto:chris.johnson3@hp.com">chris.johnson3@hp.com</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">HP Software - Security Product Group<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">(916) 785-2817<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas">----------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>