[syslog-ng] Best way to manage apache's access logs

[Nik Ambrosch]

Currently I wrote a python script that logs to disk and sends some metrics
via statsd, this only needs to be declared once for the entire apache/httpd
instance and it works for all requests, including mod_proxy, mod_jk, and
errors:

LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""
combined_virtual
CustomLog "||/usr/local/bin/vlog.py access /var/log/httpd" combined_virtual
ErrorLog "||/usr/local/bin/vlog.py error /var/log/httpd"

This has worked fine for years and there are no real problems, I've just
been thinking about a "better" way to do it.  If i can ship logs to a
centralized server *and* write to local disk then I could kill off my daily
jobs to fetch and archive access logs.

The question i'm asking is if there's a better mechanism than piping to
logger.  I use logger for other projects and while it's a very old way of
doing things it still works great -

LogFormat "blahblahblah" my_custom_log_format
CustomLog "|exec /usr/bin/logger -t apache-access my_custom_log_format
ErrorLog "|exec /usr/bin/logger -t apache-error



On Tue, Mar 26, 2019 at 10:53 AM Evan Rempel <erempel at uvic.ca> wrote:

> I agree that it is much preferred to NOT use apache's built in file
> logging. Apache can log its error log to syslog ... which is great, but it
> can not log its access log, or mod_jk log or others.
>
> We wanted the ability to be able to recreate log files at a syslog
> destination for web site analytics (which some tools require) so we wrote a
> small perl program that takes
> a couple of arguments for the log type (access, error, mod_jk etc) and for
> the site name. Then we stick them into the log with a syslog ident/tag of
> httpd and a message  of the form
>
> ${log_type}: ${site_name}: {$message}
>
> where the message is the full message produced by apache using the normal
> apache log format specification.
>
> One of the advantages of using live central logging like this is that I
> can "see/watch" all of the logs for a "site_name" when the site is being
> run on a load balanced pool of apache servers.
> The other advantage is that I can stream these to my analytics system and
> generate log files of the original apache format
>
> access_log-${site_name}.log
> error_log-${site_name}.log
>
> which can then be consumed by the analytics engine.
>
> Just my $0.02 for anyone that likes it :-)
>
> Evan
>
> On 3/26/19 7:14 AM, Nik Ambrosch wrote:
>
> I would like to replace apache's file logger entirely, using syslog-ng to
> write to both network and local disk instead of just using apache to write
> to disk and syslog-ng to write to network.
>
>
> On Tue, Mar 26, 2019 at 10:08 AM Balazs Scheidler <bazsi77 at gmail.com>
> wrote:
>
>> I would follow the files using the wildcard-file() source, possibly with
>> marking them up with apache specific name-value pairs that we extract from
>> filenames and/or content.
>>
>> On Tue, Mar 26, 2019, 06:48 Nik Ambrosch <nik at ambrosch.com wrote:
>>
>>> How are people managing their apache access logs using syslog-ng these
>>> days - still just piping through logger?  I'm looking to take over logging
>>> entirely using syslog-ng, not just reading files from disk and sending to
>>> ES or something.
>>>
>>> In the past I've used cronolog, which works fine, but I'd love the
>>> flexibility of sending the logs through syslog-ng.
>>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190326/770b5b4f/attachment.html>