Hello again, I try all the weekend to do it but there is no luck on it. I want to send all apache vhosts logs to a remote server. I read this guide: http://www.facebook.com/note.php?note_id=338493890632 But, when I create the named pipes, I cannot restart Apache. I try to send the logs files this way: file("/var/log/apache2/access.log"); file("/var/log/apache2/error.log"); But on the server-side there is any access.log and error.log files. I have lots of vhost on the client-side. How could I configure it? Thanks and best regards.
Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
I want to send all apache vhosts logs to a remote server.
I read this guide: http://www.facebook.com/note.php?note_id=338493890632
But, when I create the named pipes, I cannot restart Apache.
Any error messages? I tried the guide locally, and it worked for me. With a few corrections (like fixing the $MSGONLYn template to read $MSGONLY\n, where the \ got lost in the facebook translation, I suppose).
I try to send the logs files this way:
file("/var/log/apache2/access.log"); file("/var/log/apache2/error.log");
But on the server-side there is any access.log and error.log files.
That's expected. The server does not know what files to put these logs in, so it'll put the apache logs wherever it puts the rest of the messages.
I have lots of vhost on the client-side. How could I configure it?
One way is to use named pipes, as the guide suggests. Another way is to still log to files on the web serves, but change the LogFormat to what the guide says, and read from files. Though, error.log cannot be configured, so that'll need a bit different treatment, see below. For the access logs, you can use the LogFormat in the guide, and split them on server side, as the guide says. For error logs, you can't really do that, since it cannot be formatted on apache side, as far as I remember, so it won't contain the virtual host. So if using a single error.log, the best one can do is transfer it as-is, using the no-parse source flag on the client's syslog-ng. If you need per-host split error logs, well.. there's still a few options left: either you add them one by one as sources, and make sure there's a way the server can identify the source (eg, using no-parse, and formatting the log into syslog format in the destination, where you add the virtualhost - this way you'll have a source + destination + log block for each and every virtualhost. With syslog-ng 3.3, this can be made easier using block{}, but that's something I haven't played with much, and syslog-ng 3.3 is in beta at the moment...). Another way is to use an external program to find all the error.logs, tail -f them, and format the lines appropriately, and send them to syslog. This sounds pretty awkward, though. -- |8]
Thanks, I just watch that apache logs goes to "user" log file. This is the tree: $ tree -h host1 . âââ [4.0K] 2011-07 âââ [ 19K] auth âââ [346K] authpriv âââ [297K] cron âââ [ 12K] daemon âââ [2.1K] kern âââ [7.9K] syslog âââ [775K] user $ cat user Jul 11 13:10:10 host1/host1 IP - - [11/Jul/2011:13:10:09 +0200] "GET /favicon.ico HTTP/1.1" 404 503 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30" ... This looks great, but I want to get all vhost on different files, something like this: domain1 - > apache_access_domain1.log domain2 - > apache_access_domain2.log domain3 - > apache_access_domain3.log ... Is this possible? I must configure on the client or server side? Thanks for all your help. Best regards. -----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Gergely Nagy Enviado el: lunes, 11 de julio de 2011 12:19 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
I want to send all apache vhosts logs to a remote server.
I read this guide: http://www.facebook.com/note.php?note_id=338493890632
But, when I create the named pipes, I cannot restart Apache.
Any error messages? I tried the guide locally, and it worked for me. With a few corrections (like fixing the $MSGONLYn template to read $MSGONLY\n, where the \ got lost in the facebook translation, I suppose).
I try to send the logs files this way:
file("/var/log/apache2/access.log"); file("/var/log/apache2/error.log");
But on the server-side there is any access.log and error.log files.
That's expected. The server does not know what files to put these logs in, so it'll put the apache logs wherever it puts the rest of the messages.
I have lots of vhost on the client-side. How could I configure it?
One way is to use named pipes, as the guide suggests. Another way is to still log to files on the web serves, but change the LogFormat to what the guide says, and read from files. Though, error.log cannot be configured, so that'll need a bit different treatment, see below. For the access logs, you can use the LogFormat in the guide, and split them on server side, as the guide says. For error logs, you can't really do that, since it cannot be formatted on apache side, as far as I remember, so it won't contain the virtual host. So if using a single error.log, the best one can do is transfer it as-is, using the no-parse source flag on the client's syslog-ng. If you need per-host split error logs, well.. there's still a few options left: either you add them one by one as sources, and make sure there's a way the server can identify the source (eg, using no-parse, and formatting the log into syslog format in the destination, where you add the virtualhost - this way you'll have a source + destination + log block for each and every virtualhost. With syslog-ng 3.3, this can be made easier using block{}, but that's something I haven't played with much, and syslog-ng 3.3 is in beta at the moment...). Another way is to use an external program to find all the error.logs, tail -f them, and format the lines appropriately, and send them to syslog. This sounds pretty awkward, though. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
$ cat user Jul 11 13:10:10 host1/host1 IP - - [11/Jul/2011:13:10:09 +0200] "GET /favicon.ico HTTP/1.1" 404 503 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30" ...
This looks great, but I want to get all vhost on different files, something like this:
domain1 - > apache_access_domain1.log domain2 - > apache_access_domain2.log domain3 - > apache_access_domain3.log ...
Is this possible? I must configure on the client or server side?
Yes, it is possible. As long as the virtual host's name appears in the log message, it is possible. Something along these lines could work (assuming keep_hostname(yes)): # Here, find a pattern with which you can identify apache logs. # It's possible to alter the template on the webserver, to include a # token with chich the messages become easily identifiable: for example, # using this LogFormat (in apache): # # LogFormat "Jan 12 12:12:12 %v apache[666]: %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined filter f_apache { program("apache"); }; destination d_apache { file("/var/log/apache/apache_access_${HOST}.log"); }; log { source(s_src); filter(f_apache); destination(d_apache); }; -- |8]
On 07/11/11 10:14, Josu Lazkano wrote:
I have lots of vhost on the client-side. How could I configure it?
I find it preferable to have Apache log to syslog in the first place, e.g. with: CustomLog "||/usr/bin/logger -p local2.info -t apache" combined ErrorLog "||/usr/bin/logger -p local3.info -t apache" I use a custom log format to include a VHost field in access log lines (and later split them as needed). If you need seperate error logs as well, then it is probably best to include the VHost name in the tag/programname field. -- Martin
Hello again, I configure on the client side some vhost log files this way: $ cat syslog-ng.conf options { log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(yes); stats_freq(0); bad_hostname("^gconfd$"); }; source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg"); file("/var/www/domain1/log/access.log"); file("/var/www/domain2/log/access.log"); file("/var/www/domain3/log/access.log"); file("/var/www/domain4/log/access.log"); }; destination logserver { tcp("logserver"); }; log { source(s_all); destination(logserver); }; With this configuration I don't get all access.log info, I put on the same screen a tail -f with a local access.log and user log on the logserver and on the vhost file there is more info. Is this configuration correct? Sorry for all those newby questions. Thanks for all your help and best regards. -----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Martin Schütte Enviado el: lunes, 11 de julio de 2011 21:06 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs On 07/11/11 10:14, Josu Lazkano wrote:
I have lots of vhost on the client-side. How could I configure it?
I find it preferable to have Apache log to syslog in the first place, e.g. with: CustomLog "||/usr/bin/logger -p local2.info -t apache" combined ErrorLog "||/usr/bin/logger -p local3.info -t apache" I use a custom log format to include a VHost field in access log lines (and later split them as needed). If you need seperate error logs as well, then it is probably best to include the VHost name in the tag/programname field. -- Martin ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi! are you using "combined" or "vhost_combined" in your V.H. config? For example: CustomLog "|/usr/bin/logger -s -p local5.info " vhost_combined Bye! On 07/13/2011 06:49 AM, Josu Lazkano wrote:
Hello again, I configure on the client side some vhost log files this way:
$ cat syslog-ng.conf options { log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(yes); stats_freq(0); bad_hostname("^gconfd$"); };
source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg"); file("/var/www/domain1/log/access.log"); file("/var/www/domain2/log/access.log"); file("/var/www/domain3/log/access.log"); file("/var/www/domain4/log/access.log"); };
destination logserver { tcp("logserver"); };
log { source(s_all); destination(logserver); };
With this configuration I don't get all access.log info, I put on the same screen a tail -f with a local access.log and user log on the logserver and on the vhost file there is more info. Is this configuration correct?
Sorry for all those newby questions.
Thanks for all your help and best regards.
-----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Martin Schütte Enviado el: lunes, 11 de julio de 2011 21:06 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs
On 07/11/11 10:14, Josu Lazkano wrote:
I have lots of vhost on the client-side. How could I configure it? I find it preferable to have Apache log to syslog in the first place, e.g. with: CustomLog "||/usr/bin/logger -p local2.info -t apache" combined ErrorLog "||/usr/bin/logger -p local3.info -t apache"
I use a custom log format to include a VHost field in access log lines (and later split them as needed). If you need seperate error logs as well, then it is probably best to include the VHost name in the tag/programname field.
----- CeSPI Centro Superior para el Procesamiento de la Información Universidad Nacional de La Plata ------------------------------------------------------------------------------- Proteja el Medioambiente. No imprima este mail si no es absolutamente necesario
Thanks!!! This work for me, las question. Is possible to add a facility name on apache to archive on the logserver as "apache" file? Best regards. -----Mensaje original----- De: Matias Banchoff [mailto:matiasb@cespi.unlp.edu.ar] Enviado el: miércoles, 13 de julio de 2011 13:54 Para: Syslog-ng users' and developers' mailing list CC: Josu Lazkano Asunto: Re: [syslog-ng] Apache vhost logs Hi! are you using "combined" or "vhost_combined" in your V.H. config? For example: CustomLog "|/usr/bin/logger -s -p local5.info " vhost_combined Bye! On 07/13/2011 06:49 AM, Josu Lazkano wrote:
Hello again, I configure on the client side some vhost log files this way:
$ cat syslog-ng.conf options { log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(yes); stats_freq(0); bad_hostname("^gconfd$"); };
source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg"); file("/var/www/domain1/log/access.log"); file("/var/www/domain2/log/access.log"); file("/var/www/domain3/log/access.log"); file("/var/www/domain4/log/access.log"); };
destination logserver { tcp("logserver"); };
log { source(s_all); destination(logserver); };
With this configuration I don't get all access.log info, I put on the same screen a tail -f with a local access.log and user log on the logserver and on the vhost file there is more info. Is this configuration correct?
Sorry for all those newby questions.
Thanks for all your help and best regards.
-----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Martin Schütte Enviado el: lunes, 11 de julio de 2011 21:06 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs
On 07/11/11 10:14, Josu Lazkano wrote:
I have lots of vhost on the client-side. How could I configure it? I find it preferable to have Apache log to syslog in the first place, e.g. with: CustomLog "||/usr/bin/logger -p local2.info -t apache" combined ErrorLog "||/usr/bin/logger -p local3.info -t apache"
I use a custom log format to include a VHost field in access log lines (and later split them as needed). If you need seperate error logs as well, then it is probably best to include the VHost name in the tag/programname field.
----- CeSPI Centro Superior para el Procesamiento de la Información Universidad Nacional de La Plata ------------------------------------------------------------------------------- Proteja el Medioambiente. No imprima este mail si no es absolutamente necesario
Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
Thanks!!!
This work for me, las question.
Is possible to add a facility name on apache to archive on the logserver as "apache" file?
Facilities are transmitted as numbers over the wire. Using the CustomLog suggested by Matias, you could do something like this on server side: filter f_local5 { facility(local5); }; destination d_apache { file("/var/log/apache.log"); }; log { source(s_all); filter(f_local5); destination(d_apache); }; (Not tested, but should work nevertheless) -- |8]
Thanks again, I need to change a little changes. I configure this way: filter f_local5 { facility(local5); }; destination d_apache { file("/var/log/apache.log"); }; log { source(s_all); filter(f_local5); destination(d_apache); }; But I want to save the apache logs on a directory inside each host: destination extern { file("/var/log/extern/${HOST}/${YEAR}-${MONTH}/${FACILITY}"); }; I wan tto change the name "local5" to "apache" $ tree /var/log/extern/host1/ /var/log/extern/fbmwebs01/ âââ 2011-07 âââ auth âââ authpriv âââ cron âââ daemon âââ kern âââ local5 <-- I want to change this âââ syslog âââ user This is all the configuration on the server side: options { log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(yes); stats_freq(0); bad_hostname("^gconfd$"); }; source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); tcp(); }; destination extern { file("/var/log/extern/${HOST}/${YEAR}-${MONTH}/${FACILITY}"); }; log { source(s_all); destination(extern); }; filter f_local5 { facility(local5); }; destination d_apache { file("/var/log/extern/apache.log"); }; log { source(s_all); filter(f_local5); destination(d_apache); }; Thanks for your great help, best regards. -----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Gergely Nagy Enviado el: jueves, 14 de julio de 2011 11:40 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
Thanks!!!
This work for me, las question.
Is possible to add a facility name on apache to archive on the logserver as "apache" file?
Facilities are transmitted as numbers over the wire. Using the CustomLog suggested by Matias, you could do something like this on server side: filter f_local5 { facility(local5); }; destination d_apache { file("/var/log/apache.log"); }; log { source(s_all); filter(f_local5); destination(d_apache); }; (Not tested, but should work nevertheless) -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
filter f_local5 { facility(local5); };
destination d_apache { file("/var/log/apache.log"); };
log { source(s_all); filter(f_local5); destination(d_apache); };
But I want to save the apache logs on a directory inside each host:
destination extern { file("/var/log/extern/${HOST}/${YEAR}-${MONTH}/${FACILITY}"); };
I wan tto change the name "local5" to "apache"
That's not possible like this. However, since you already filter on f_local5, you can add another destination, along side extern: destination d_apache { file("/var/log/extern/${HOST}/${YEAR}-${MONTH}/apache"); }; (This would replace the d_apache you have in your configs now) -- |8]
Perfect! The last last question, is it posible to NOT write on local5 file? I have same info on the apache and local5 files. Best regards. -----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Gergely Nagy Enviado el: jueves, 14 de julio de 2011 12:06 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
filter f_local5 { facility(local5); };
destination d_apache { file("/var/log/apache.log"); };
log { source(s_all); filter(f_local5); destination(d_apache); };
But I want to save the apache logs on a directory inside each host:
destination extern { file("/var/log/extern/${HOST}/${YEAR}-${MONTH}/${FACILITY}"); };
I wan tto change the name "local5" to "apache"
That's not possible like this. However, since you already filter on f_local5, you can add another destination, along side extern: destination d_apache { file("/var/log/extern/${HOST}/${YEAR}-${MONTH}/apache"); }; (This would replace the d_apache you have in your configs now) -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
Perfect!
The last last question, is it posible to NOT write on local5 file?
I have same info on the apache and local5 files.
Yes. Make the log{} block with the d_apache destination the first log{} block in your config, and add a flags(final) flag to it: log { source(s_all); filter(f_local5); destination(d_apache); flags(final); }; See the relevant section of the documentation here: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... -- |8]
Thanks!!! With This configuration it Works great: log { source(s_all); filter(f_local5); destination(d_apache); flags(final); }; log { source(s_all); destination(extern); }; Best regards. -----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Gergely Nagy Enviado el: jueves, 14 de julio de 2011 12:29 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs Josu Lazkano <josu.lazkano@barcelonamedia.org> writes:
Perfect!
The last last question, is it posible to NOT write on local5 file?
I have same info on the apache and local5 files.
Yes. Make the log{} block with the d_apache destination the first log{} block in your config, and add a flags(final) flag to it: log { source(s_all); filter(f_local5); destination(d_apache); flags(final); }; See the relevant section of the documentation here: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Thursday, 14 July 2011 11:34:28 Josu Lazkano wrote:
Thanks!!!
This work for me, las question.
Is possible to add a facility name on apache to archive on the logserver as "apache" file?
Facility names are hard-coded at the kernel level, the only ones you can have are those in "man 3 syslog". You can't add new ones. So you have to use some other way to identify your apache logs. A good way is to pick an unused local0 to local7 facility (watch out for two hosts using the same local for different things and sending them to the same syslogger!)
Best regards.
-----Mensaje original----- De: Matias Banchoff [mailto:matiasb@cespi.unlp.edu.ar] Enviado el: miércoles, 13 de julio de 2011 13:54 Para: Syslog-ng users' and developers' mailing list CC: Josu Lazkano Asunto: Re: [syslog-ng] Apache vhost logs
Hi! are you using "combined" or "vhost_combined" in your V.H. config? For example: CustomLog "|/usr/bin/logger -s -p local5.info " vhost_combined Bye!
On 07/13/2011 06:49 AM, Josu Lazkano wrote:
Hello again, I configure on the client side some vhost log files this way:
$ cat syslog-ng.conf options {
log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(yes); stats_freq(0); bad_hostname("^gconfd$");
};
source s_all {
internal(); unix-stream("/dev/log"); file("/proc/kmsg"); file("/var/www/domain1/log/access.log"); file("/var/www/domain2/log/access.log"); file("/var/www/domain3/log/access.log"); file("/var/www/domain4/log/access.log");
};
destination logserver {
tcp("logserver");
};
log {
source(s_all); destination(logserver);
};
With this configuration I don't get all access.log info, I put on the same screen a tail -f with a local access.log and user log on the logserver and on the vhost file there is more info. Is this configuration correct?
Sorry for all those newby questions.
Thanks for all your help and best regards.
-----Mensaje original----- De: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Martin Schütte Enviado el: lunes, 11 de julio de 2011 21:06 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Apache vhost logs
On 07/11/11 10:14, Josu Lazkano wrote:
I have lots of vhost on the client-side. How could I configure it?
I find it preferable to have Apache log to syslog in the first place, e.g. with: CustomLog "||/usr/bin/logger -p local2.info -t apache" combined ErrorLog "||/usr/bin/logger -p local3.info -t apache"
I use a custom log format to include a VHost field in access log lines (and later split them as needed). If you need seperate error logs as well, then it is probably best to include the VHost name in the tag/programname field.
----- CeSPI Centro Superior para el Procesamiento de la Información
Universidad Nacional de La Plata -------------------------------------------------------------------- ----------- Proteja el Medioambiente. No imprima este mail si no es absolutamente necesario ___________________________________________________________________ ___________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions +27 11 575 7585 Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you.
participants (5)
-
Alan McKinnon
-
Gergely Nagy
-
Josu Lazkano
-
Martin Schütte
-
Matias Banchoff