Best way to manage apache's access logs
How are people managing their apache access logs using syslog-ng these days - still just piping through logger? I'm looking to take over logging entirely using syslog-ng, not just reading files from disk and sending to ES or something. In the past I've used cronolog, which works fine, but I'd love the flexibility of sending the logs through syslog-ng.
I would follow the files using the wildcard-file() source, possibly with marking them up with apache specific name-value pairs that we extract from filenames and/or content. On Tue, Mar 26, 2019, 06:48 Nik Ambrosch <nik@ambrosch.com wrote:
How are people managing their apache access logs using syslog-ng these days - still just piping through logger? I'm looking to take over logging entirely using syslog-ng, not just reading files from disk and sending to ES or something.
In the past I've used cronolog, which works fine, but I'd love the flexibility of sending the logs through syslog-ng.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I would like to replace apache's file logger entirely, using syslog-ng to write to both network and local disk instead of just using apache to write to disk and syslog-ng to write to network. On Tue, Mar 26, 2019 at 10:08 AM Balazs Scheidler <bazsi77@gmail.com> wrote:
I would follow the files using the wildcard-file() source, possibly with marking them up with apache specific name-value pairs that we extract from filenames and/or content.
On Tue, Mar 26, 2019, 06:48 Nik Ambrosch <nik@ambrosch.com wrote:
How are people managing their apache access logs using syslog-ng these days - still just piping through logger? I'm looking to take over logging entirely using syslog-ng, not just reading files from disk and sending to ES or something.
In the past I've used cronolog, which works fine, but I'd love the flexibility of sending the logs through syslog-ng.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I agree that it is much preferred to NOT use apache's built in file logging. Apache can log its error log to syslog ... which is great, but it can not log its access log, or mod_jk log or others. We wanted the ability to be able to recreate log files at a syslog destination for web site analytics (which some tools require) so we wrote a small perl program that takes a couple of arguments for the log type (access, error, mod_jk etc) and for the site name. Then we stick them into the log with a syslog ident/tag of httpd and a message of the form ${log_type}: ${site_name}: {$message} where the message is the full message produced by apache using the normal apache log format specification. One of the advantages of using live central logging like this is that I can "see/watch" all of the logs for a "site_name" when the site is being run on a load balanced pool of apache servers. The other advantage is that I can stream these to my analytics system and generate log files of the original apache format access_log-${site_name}.log error_log-${site_name}.log which can then be consumed by the analytics engine. Just my $0.02 for anyone that likes it :-) Evan On 3/26/19 7:14 AM, Nik Ambrosch wrote:
I would like to replace apache's file logger entirely, using syslog-ng to write to both network and local disk instead of just using apache to write to disk and syslog-ng to write to network.
On Tue, Mar 26, 2019 at 10:08 AM Balazs Scheidler <bazsi77@gmail.com <mailto:bazsi77@gmail.com>> wrote:
I would follow the files using the wildcard-file() source, possibly with marking them up with apache specific name-value pairs that we extract from filenames and/or content.
On Tue, Mar 26, 2019, 06:48 Nik Ambrosch <nik@ambrosch.com <mailto:nik@ambrosch.com> wrote:
How are people managing their apache access logs using syslog-ng these days - still just piping through logger? I'm looking to take over logging entirely using syslog-ng, not just reading files from disk and sending to ES or something.
In the past I've used cronolog, which works fine, but I'd love the flexibility of sending the logs through syslog-ng.
Currently I wrote a python script that logs to disk and sends some metrics via statsd, this only needs to be declared once for the entire apache/httpd instance and it works for all requests, including mod_proxy, mod_jk, and errors: LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined_virtual CustomLog "||/usr/local/bin/vlog.py access /var/log/httpd" combined_virtual ErrorLog "||/usr/local/bin/vlog.py error /var/log/httpd" This has worked fine for years and there are no real problems, I've just been thinking about a "better" way to do it. If i can ship logs to a centralized server *and* write to local disk then I could kill off my daily jobs to fetch and archive access logs. The question i'm asking is if there's a better mechanism than piping to logger. I use logger for other projects and while it's a very old way of doing things it still works great - LogFormat "blahblahblah" my_custom_log_format CustomLog "|exec /usr/bin/logger -t apache-access my_custom_log_format ErrorLog "|exec /usr/bin/logger -t apache-error On Tue, Mar 26, 2019 at 10:53 AM Evan Rempel <erempel@uvic.ca> wrote:
I agree that it is much preferred to NOT use apache's built in file logging. Apache can log its error log to syslog ... which is great, but it can not log its access log, or mod_jk log or others.
We wanted the ability to be able to recreate log files at a syslog destination for web site analytics (which some tools require) so we wrote a small perl program that takes a couple of arguments for the log type (access, error, mod_jk etc) and for the site name. Then we stick them into the log with a syslog ident/tag of httpd and a message of the form
${log_type}: ${site_name}: {$message}
where the message is the full message produced by apache using the normal apache log format specification.
One of the advantages of using live central logging like this is that I can "see/watch" all of the logs for a "site_name" when the site is being run on a load balanced pool of apache servers. The other advantage is that I can stream these to my analytics system and generate log files of the original apache format
access_log-${site_name}.log error_log-${site_name}.log
which can then be consumed by the analytics engine.
Just my $0.02 for anyone that likes it :-)
Evan
On 3/26/19 7:14 AM, Nik Ambrosch wrote:
I would like to replace apache's file logger entirely, using syslog-ng to write to both network and local disk instead of just using apache to write to disk and syslog-ng to write to network.
On Tue, Mar 26, 2019 at 10:08 AM Balazs Scheidler <bazsi77@gmail.com> wrote:
I would follow the files using the wildcard-file() source, possibly with marking them up with apache specific name-value pairs that we extract from filenames and/or content.
On Tue, Mar 26, 2019, 06:48 Nik Ambrosch <nik@ambrosch.com wrote:
How are people managing their apache access logs using syslog-ng these days - still just piping through logger? I'm looking to take over logging entirely using syslog-ng, not just reading files from disk and sending to ES or something.
In the past I've used cronolog, which works fine, but I'd love the flexibility of sending the logs through syslog-ng.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Balazs Scheidler
-
Evan Rempel
-
Nik Ambrosch