[syslog-ng]Apache Access Log Load and Syslog-ng Stability
Jason Stafford
syslog-ng@lists.balabit.hu
Thu, 10 Mar 2005 21:09:18 -0600
This is a multi-part message in MIME format.
------=_NextPart_000_0157_01C525B5.6C38D7A0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Howdy,
I'm trying to setup a load balanced system with several web servers and =
one central log server. Currently I have syslog-ng running and pipe the =
output form apache to logger. This is all running locally on one box. =
However, syslog-ng just randomly hangs and apache stop processing =
requests. I can restart apache and it still does not respond, but i can =
just restart syslog-ng and then it all starts working again. Is =
syslog-ng designed to handle this kinda of load, around 3 million =
entries a day? I have been googling for hours and can only find some =
comments about log_fifo_size and reap_time in the options section? =20
here is my current config, if that helps any
options { sync (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
};
destination d_http_access { =
file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.accesslog" =
owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(\
yes)); };
destination d_http_search { =
file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.searchlog" =
owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(\
yes)); };
destination d_http_error { =
file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.errorlog" =
owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(ye\
s)); };
filter f_http_access { match("APACHE_ACCESS_LOG") and
not match("test.html"); };
filter f_http_search { match("APACHE_ACCESS_LOG") and
match("/content/search"); };
filter f_http_error { match("APACHE_ERR_LOG"); };
log { source(s_sys); filter(f_http_access); destination(d_http_access); =
};
#Log httpd search access logs
log { source(s_sys); filter(f_http_search); destination(d_http_search); =
};
#Log httpd error logs
log { source(s_sys); filter(f_http_error); destination(d_http_error); };
###################################################3
httpd.conf log line
CustomLog "|/usr/bin/logger -p local0.info -t APACHE_ACCESS_LOG" tracking
------=_NextPart_000_0157_01C525B5.6C38D7A0
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2800.1491" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>Howdy,</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>I'm trying to setup a load balanced =
system with=20
several web servers and one central log server. Currently I have =
syslog-ng=20
running and pipe the output form apache to logger. This is all running =
locally=20
on one box. However, syslog-ng just randomly hangs and apache stop =
processing requests. I can restart apache and it still does not respond, =
but i=20
can just restart syslog-ng and then it all starts working again. =
Is=20
syslog-ng designed to handle this kinda of load, around 3 million =
entries a day?=20
I have been googling for hours and can only find some comments=20
about log_fifo_size and reap_time in the options =
section? =20
</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>here is my current config, if that =
helps=20
any</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>options { sync=20
(0);<BR> =
time_reopen=20
(10);<BR> =
log_fifo_size=20
(1000);<BR> =
long_hostnames=20
(off);<BR> use_dns =
(no);<BR> use_fqdn =
(no);<BR> =
create_dirs=20
(no);<BR> =
keep_hostname=20
(yes);<BR> };<BR></FONT></DIV>
<DIV><FONT face=3DArial size=3D2>destination d_http_access {=20
file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.accesslog" =
owner(root)=20
group(root) perm(0600) dir_perm(0700) create_dirs(\<BR>yes)); =
};<BR>destination=20
d_http_search {=20
file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.searchlog" =
owner(root)=20
group(root) perm(0600) dir_perm(0700) create_dirs(\<BR>yes)); =
};<BR>destination=20
d_http_error { =
file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.errorlog"=20
owner(root) group(root) perm(0600) dir_perm(0700) =
create_dirs(ye\<BR>s));=20
};<BR></DIV></FONT>
<DIV><FONT face=3DArial size=3D2>filter =
f_http_access {=20
match("APACHE_ACCESS_LOG")=20
and<BR> =
=
not match("test.html"); };<BR>filter =
f_http_search {=20
match("APACHE_ACCESS_LOG")=20
and<BR> =
=
match("/content/search"); };<BR>filter =
f_http_error {=20
match("APACHE_ERR_LOG"); };<BR></FONT></DIV>
<DIV><FONT face=3DArial size=3D2>log { source(s_sys); =
filter(f_http_access);=20
destination(d_http_access); };<BR>#Log httpd search access logs<BR>log { =
source(s_sys); filter(f_http_search); destination(d_http_search); =
};<BR>#Log=20
httpd error logs<BR>log { source(s_sys); filter(f_http_error);=20
destination(d_http_error); };<BR></DIV></FONT>
<DIV><FONT face=3DArial=20
size=3D2>###################################################3</FONT></DIV=
>
<DIV><FONT face=3DArial size=3D2>httpd.conf log line</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>CustomLog "|/usr/bin/logger -p =
local0.info -t=20
APACHE_ACCESS_LOG" tracking</DIV></FONT></BODY></HTML>
------=_NextPart_000_0157_01C525B5.6C38D7A0--