[syslog-ng]Disk error handled properly?
John Feeney
syslog-ng@lists.balabit.hu
Fri, 25 Jun 2004 17:14:36 -0400
This is a multi-part message in MIME format.
------_=_NextPart_001_01C45AF9.6AF346B8
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hello,
=20
I am running syslog-ng (1.4.17) on Linux (2.4.21) and received an error=20
when the disk partition became full. Looking at the code, it does not
look=20
as though the error was handled very well. After the error, syslog-ng=20
started to take up 26% of my CPU time and was unresponsive to other=20
logged messages to the file that was being written to when the error=20
occurred (even though the room was made in the partition).
=20
The specifics are:
=20
Received this error -=20
Jun 25 13:16:57 port1-1 syslog-ng[147]:=20
io.c: do_write: write() failed (errno 28),=20
No space left on device
=20
So, do_write() correctly identifies the error and puts out a nice=20
message that the disk is full.
=20
int res =3D write(self->fd, data, length);
if(res < 0)
{
switch(errno)
{
case EINTR:
case EWOULDBLOCK:
return 0;
default:
werror(
"io.c: do_write: write() failed (errno %i), %z\n");
}
...
return(res);
}
=20
The only place I can see where do_write is called is write_callback()=20
where I would have liked to have seen a "if(w < -1) do_something" but=20
the value of w is just sent in as an argument to BUF_FLUSH().
=20
static void write_callback(struct nonblocking_fd *fd)
{=20
CAST(io_fd, self, fd);
int res;
=20
struct fd_write w =3D
{{ STACK_HEADER, do_write), fd->fd, self->fsync };
=20
assert(self->buffer);
=20
res =3D BUF_FLUSH(self->buffer, &w.super);
=20
So, if w is an error (-1), what is &w.super equal to? Something not=20
good.
=20
Maybe I have this wrong. Maybe there is another place where
do_write() is called. Regardless, the value of w is not being checked=20
so what is the proper course of action if w is -1?
=20
Note: my syslog-ng is running on an embedded system that does
not have much flash memory to store these logs. The system has=20
a process that truncates the log files when they exceed specific limits,
but the granularity of the file size checking can mean that one or more=20
processes can exceed their limit(s) and cause the disk to be full before
the checker process wakes up and does the cleanup, as happened in=20
this case.
=20
Thanks.
John Feeney
=20
--------------------------------------------------------
=20
This email message and any files transmitted with it contain =
confidential information intended only for the person(s) to whom this =
email message is addressed. If you have received this email message in =
error, please notify the sender immediately by telephone or email and =
destroy the original message without making a copy. Thank you.=20
=20
------_=_NextPart_001_01C45AF9.6AF346B8
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<HTML xmlns:eXclaimer=3D"http://www.exclaimer.co.uk">
<HEAD>
<META http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3DUTF-16">
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3DUTF-16">
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 10 (filtered)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</HEAD><BODY lang=3DEN-US link=3Dblue vlink=3Dpurple><DIV>
<div class=3DSection1>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Hello,</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I am running syslog-ng (1.4.17) on Linux (2.4.21) and
received an error </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>when the disk partition became full. Looking at the =
code, it
does not look </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>as though the error was handled very well. After the =
error, syslog-ng
</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>started to take up 26% of my CPU time and was =
unresponsive
to other </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>logged messages to the file that was being written to =
when
the error </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>occurred (even though the room was made in the =
partition).</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>The specifics are:</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> Received this error – =
</span></font></p>
<p class=3DMsoNormal><b><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:
10.0pt;font-family:"Courier New";font-weight:bold'>Jun 25 =
</span></font></b><b><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt;font-family:"Courier New";
font-weight:bold'>13:16:57</span></font></b><b><font size=3D2 =
face=3D"Courier New"><span
style=3D'font-size:10.0pt;font-family:"Courier New";font-weight:bold'> =
port1-1 syslog-ng[147]:
</span></font></b></p>
<p class=3DMsoNormal><b><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:
10.0pt;font-family:"Courier New";font-weight:bold'> io.c: =
do_write: write()
failed (errno 28), </span></font></b></p>
<p class=3DMsoNormal><b><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:
10.0pt;font-family:"Courier New";font-weight:bold'> No space =
left on device</span></font></b></p>
<p class=3DMsoNormal><b><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:
10.0pt;font-family:"Courier =
New";font-weight:bold'> </span></font></b></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>So, do_write() correctly identifies the error and =
puts out a
nice </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>message that the disk is full.</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> int res =3D write(self->fd, data, =
length);</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> if(res < 0)</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> {</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> switch(errno)</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> {</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> case =
EINTR:</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> case =
EWOULDBLOCK:</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> =
return 0;</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> =
default:</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> =
werror(</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> =
“io.c: do_write: write() failed (errno %i),
%z\n”);</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> }</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> ...</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> return(res);</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> }</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>The only place I can see where do_write is called is =
write_callback()
</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>where I would have liked to have seen a “if(w =
< -1)
do_something” but </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>the value of w is just sent in as an argument to =
BUF_FLUSH().</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>static void write_callback(struct nonblocking_fd =
*fd)</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>{ </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> CAST(io_fd, self, fd);</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> int res;</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> struct fd_write <b><span =
style=3D'font-weight:bold'>w</span></b>
=3D</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> {{ STACK_HEADER, do_write), =
fd->fd, self->fsync };</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> =
assert(self->buffer);</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> res =3D BUF_FLUSH(self->buffer, =
<b><span
style=3D'font-weight:bold'>&w.super</span></b>);</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>So, if w is an error (-1), what is &w.super equal =
to? Something
not </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>good.</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Maybe I have this wrong. Maybe there is another place =
where</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>do_write() is called. Regardless, the value of w is =
not
being checked </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>so what is the proper course of action if w is =
-1?</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Note: my syslog-ng is running on an embedded system =
that
does</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>not have much flash memory to store these logs. The =
system
has </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>a process that truncates the log files when they =
exceed
specific limits, </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>but the granularity of the file size checking can =
mean that
one or more </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>processes can exceed their limit(s) and cause the =
disk to be
full before </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>the checker process wakes up and does the cleanup, as =
happened
in </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>this case.</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> </span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Thanks.</span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'> John Feeney</span></font></p>
</div>
</DIV>
<DIV>
<HR>
</DIV>
<DIV STYLE=3D"FONT-SIZE: 7pt; COLOR: gray; FONT-FAMILY: verdana">This =
email message and any files transmitted with it contain confidential =
information intended only for the person(s) to whom this email message =
is addressed. If you have received this email message in error, please =
notify the sender immediately by telephone or email and destroy the =
original message without making a copy. Thank you. </DIV></BODY></HTML>
------_=_NextPart_001_01C45AF9.6AF346B8--