Apache Access Log Load and Syslog-ng Stability
Howdy, I'm trying to setup a load balanced system with several web servers and one central log server. Currently I have syslog-ng running and pipe the output form apache to logger. This is all running locally on one box. However, syslog-ng just randomly hangs and apache stop processing requests. I can restart apache and it still does not respond, but i can just restart syslog-ng and then it all starts working again. Is syslog-ng designed to handle this kinda of load, around 3 million entries a day? I have been googling for hours and can only find some comments about log_fifo_size and reap_time in the options section? here is my current config, if that helps any options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_http_access { file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.accesslog" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(\ yes)); }; destination d_http_search { file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.searchlog" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(\ yes)); }; destination d_http_error { file("/var/log/HTTP/$YEAR-$MONTH-$DAY/$YEAR-$MONTH-$DAY.errorlog" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(ye\ s)); }; filter f_http_access { match("APACHE_ACCESS_LOG") and not match("test.html"); }; filter f_http_search { match("APACHE_ACCESS_LOG") and match("/content/search"); }; filter f_http_error { match("APACHE_ERR_LOG"); }; log { source(s_sys); filter(f_http_access); destination(d_http_access); }; #Log httpd search access logs log { source(s_sys); filter(f_http_search); destination(d_http_search); }; #Log httpd error logs log { source(s_sys); filter(f_http_error); destination(d_http_error); }; ###################################################3 httpd.conf log line CustomLog "|/usr/bin/logger -p local0.info -t APACHE_ACCESS_LOG" tracking
On Thu, 10 Mar 2005 21:09:18 -0600, Jason Stafford <jason@dpcomm.net> wrote:
I'm trying to setup a load balanced system with several web servers and one central log server. Currently I have syslog-ng running and pipe the output form apache to logger. This is all running locally on one box. However, syslog-ng just randomly hangs and apache stop processing requests.
This sounds like an issue with the pipe. If you move the logger off-box and switch to UDP, you won't have the problem with Apache hanging, if the remote logger dies, then you just won't have log data.
I can restart apache and it still does not respond, but i can just restart syslog-ng and then it all starts working again. Is syslog-ng designed to handle this kinda of load, around 3 million entries a day?
I have a central logger doing 5 million entries an hour, so I'd say yes, syslog-ng can handle this load :) What sources does syslog-ng accept from on the central logger? TCP? UDP? both? local socket/pipe? I've never used syslog-ng with a local log socket or pipe.
I have been googling for hours and can only find some comments about log_fifo_size and reap_time in the options section?
Currently I have the following: sync(256); log_fifo_size(4095); time_reap(300); log_msg_size(2048);
here is my current config, if that helps any
options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; . . . httpd.conf log line CustomLog "|/usr/bin/logger -p local0.info -t APACHE_ACCESS_LOG" tracking
I do not recall if this is the recommended way to have apache log to syslog? Kevin
----- Original Message ----- From: "Kevin" <kkadow@gmail.com> To: <syslog-ng@lists.balabit.hu> Sent: Thursday, March 10, 2005 10:15 PM Subject: Re: [syslog-ng]Apache Access Log Load and Syslog-ng Stability
On Thu, 10 Mar 2005 21:09:18 -0600, Jason Stafford <jason@dpcomm.net> wrote:
I'm trying to setup a load balanced system with several web servers and one central log server. Currently I have syslog-ng running and pipe the output form apache to logger. This is all running locally on one box. However, syslog-ng just randomly hangs and apache stop processing requests.
This sounds like an issue with the pipe. If you move the logger off-box and switch to UDP, you won't have the problem with Apache hanging, if the remote logger dies, then you just won't have log data.
I can restart apache and it still does not respond, but i can just restart syslog-ng and then it all starts working again. Is syslog-ng designed to handle this kinda of load, around 3 million entries a day?
I have a central logger doing 5 million entries an hour, so I'd say yes, syslog-ng can handle this load :) That's great news, i was really hoping that wasn't the problem.
What sources does syslog-ng accept from on the central logger? TCP? UDP? both? local socket/pipe?
Each webserver is currently setup with tcp and a local pipe. However, i send the output from apache with a pipe, then setup two destination inside the syslong-ng config file, one local and the other over TCP to the central log server. Howerver, I can do away with the local pipe and switch to udp if it will fix it.
I've never used syslog-ng with a local log socket or pipe.
Could you please provide me with an example of how you get apache apache to send logs to syslog-ng, the only thing i could find was using a pipe like below.
I have been googling for hours and can only find some comments about log_fifo_size and reap_time in the options section?
Currently I have the following: sync(256); log_fifo_size(4095); time_reap(300); log_msg_size(2048);
here is my current config, if that helps any
options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; . . . httpd.conf log line CustomLog "|/usr/bin/logger -p local0.info -t APACHE_ACCESS_LOG"
tracking
I do not recall if this is the recommended way to have apache log to
syslog? any suggestions would be most welcome
Kevin
Thanks for the reply
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (2)
-
Jason Stafford
-
Kevin