[syslog-ng] New user trying to filter / rewrite apache logs

[Fekete, Róbert]

Hi, in OSE 3.9 and later there is a dedicated apache parser:
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/apache-access-log-parser.html

You might want to try it.

HTH,

Robert

On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <cifali.filipe at gmail.com>
wrote:

> Hi all,
>
> reading the docs I got into this config:
>
> source s_apache_access_log {
>
>
>     file(
>
>
>         "/var/logs/apache2/access_log"
>
>
>         follow-freq(1)
>
>
>         flags(no-parse)
>
>
>     );
>
>
> };
>
>
>
>
>
> filter f_apache_access_log {
>
>
>     match(
>
>
>         '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-
> 9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*)
> \"-\" (.*)'
>         type("pcre")
>
>
>         flags("store-matches")
>
>
>     );
>
>
> };
>
>
>
>
>
> rewrite r_apache_access_log {
>
>
>     set("$1", value("DOMAIN") condition(filter(f_apache_
> access_log)));
>
>     set("$2", value("IP") condition(filter(f_apache_
> access_log)));
>
>     set("$3", value("HTTP_METHOD") condition(filter(f_apache_
> access_log)));
>
>     set("$4", value("URI") condition(filter(f_apache_
> access_log)));
>
>     set("$6", value("HTTP_STATUS") condition(filter(f_apache_
> access_log)));
>
>     set("$7", value("SIZE") condition(filter(f_apache_
> access_log)));
>
>     set("$8", value("USER_AGENT") condition(filter(f_apache_
> access_log)));
>
> };
>
>
>
>
>
> destination d_apache_access_log {
>
>
>     mongodb(
>
>
>         # https://docs.mongodb.com/manual/reference/connection-string/
>
>
>         persist-name("apache-access-logs")
>
>
>         uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&
> socketTimeoutMS=60000&connectTimeoutMS=60000")
>
>         collection("logs")
>
>
>         retries(3600)
>
>
>         value-pairs(
>
>
>             pair("HOST", "${HOST}")
>
>
>             pair("SERVICE", "APACHE")
>
>
>             pair("DATE", "${DAY}/${MONTH}/${YEAR}")
>
>
>             pair("TIME", "${HOUR}:${MIN}")
>
>
>             pair("MESSAGE", "${MESSAGE}")
>
>
>             pair("DOMAIN", "${DOMAIN}")
>
>
>             pair("HTTP_STATUS", "${HTTP_STATUS}")
>
>
>             pair("HTTP_METHOD", "${HTTP_METHOD}")
>
>
>             pair("USER_AGENT", "${USER_AGENT}")
>
>
>             pair("SIZE", "${SIZE}")
>
>
>             pair("URI", "${URI}")
>
>
>             pair("IP", "${IP}")
>
>
>         )
>
>
>     );
>
>
> };
>
>
>
>
>
> log {
>
>
>     source(s_apache_access_log);
>
>
>     filter(f_apache_access_log);
>
>
>     rewrite(r_apache_access_log);
>
>
>     destination(d_apache_access_log);
>
>
> };
>
>
>
> but I think something is not ok, I'm not sure this is the right way to do
> it.
>
> This log produces an strange behavior:
>
> www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1"
> 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
> rv:54.0) Gecko/20100101 Firefox/54.0"
>
> but this one doesn't
>
> cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200
> 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0)
> Gecko/20100101 Firefox/54.0"
>
> The behavior is (only for subdomains):
>
> DOMAIN: ': www.cifa.li'
>
> corret one
>
> DOMAIN: 'www.cifa.li'
>
> The subdomain seems like it's adding stuff that I didn't (or want) to add.
>
>
> Am I missing something?
>
> Thanks in advance.
>
>
> --
> [ ]'s
>
> Filipe Cifali Stangler
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170711/793ead4d/attachment-0001.html>