[syslog-ng] New user trying to filter / rewrite apache logs

[Filipe Cifali]

Hi Robert,

I'm on 3.9.1, I have just tried hat example and it returns:

Error parsing affile, Error compiling template, error=Invalid template
function reference, missing function name or inbalanced '(', error_pos='24'
in /etc/syslog-ng/conf.d/apache.conf at line 54, column 18:

included from /etc/syslog-ng/syslog-ng.conf line 68, column 1

        template("$(format-json .apache.*\n"));
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

Besides the parser itself, the strange part is why the regex returning that
extra info at all...


On Tue, Jul 11, 2017 at 10:59 AM, Fekete, Róbert <robert.fekete at balabit.com>
wrote:

> Hi, in OSE 3.9 and later there is a dedicated apache parser:
> https://www.balabit.com/documents/syslog-ng-ose-
> latest-guides/en/syslog-ng-ose-guide-admin/html/apache-
> access-log-parser.html
>
> You might want to try it.
>
> HTH,
>
> Robert
>
> On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <cifali.filipe at gmail.com>
> wrote:
>
>> Hi all,
>>
>> reading the docs I got into this config:
>>
>> source s_apache_access_log {
>>
>>
>>     file(
>>
>>
>>         "/var/logs/apache2/access_log"
>>
>>
>>         follow-freq(1)
>>
>>
>>         flags(no-parse)
>>
>>
>>     );
>>
>>
>> };
>>
>>
>>
>>
>>
>> filter f_apache_access_log {
>>
>>
>>     match(
>>
>>
>>         '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-
>> 9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*)
>> \"-\" (.*)'
>>         type("pcre")
>>
>>
>>         flags("store-matches")
>>
>>
>>     );
>>
>>
>> };
>>
>>
>>
>>
>>
>> rewrite r_apache_access_log {
>>
>>
>>     set("$1", value("DOMAIN") condition(filter(f_apache_acce
>> ss_log)));
>>
>>     set("$2", value("IP") condition(filter(f_apache_acce
>> ss_log)));
>>
>>     set("$3", value("HTTP_METHOD") condition(filter(f_apache_acce
>> ss_log)));
>>
>>     set("$4", value("URI") condition(filter(f_apache_acce
>> ss_log)));
>>
>>     set("$6", value("HTTP_STATUS") condition(filter(f_apache_acce
>> ss_log)));
>>
>>     set("$7", value("SIZE") condition(filter(f_apache_acce
>> ss_log)));
>>
>>     set("$8", value("USER_AGENT") condition(filter(f_apache_acce
>> ss_log)));
>>
>> };
>>
>>
>>
>>
>>
>> destination d_apache_access_log {
>>
>>
>>     mongodb(
>>
>>
>>         # https://docs.mongodb.com/manual/reference/connection-string/
>>
>>
>>         persist-name("apache-access-logs")
>>
>>
>>         uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&sock
>> etTimeoutMS=60000&connectTimeoutMS=60000")
>>
>>         collection("logs")
>>
>>
>>         retries(3600)
>>
>>
>>         value-pairs(
>>
>>
>>             pair("HOST", "${HOST}")
>>
>>
>>             pair("SERVICE", "APACHE")
>>
>>
>>             pair("DATE", "${DAY}/${MONTH}/${YEAR}")
>>
>>
>>             pair("TIME", "${HOUR}:${MIN}")
>>
>>
>>             pair("MESSAGE", "${MESSAGE}")
>>
>>
>>             pair("DOMAIN", "${DOMAIN}")
>>
>>
>>             pair("HTTP_STATUS", "${HTTP_STATUS}")
>>
>>
>>             pair("HTTP_METHOD", "${HTTP_METHOD}")
>>
>>
>>             pair("USER_AGENT", "${USER_AGENT}")
>>
>>
>>             pair("SIZE", "${SIZE}")
>>
>>
>>             pair("URI", "${URI}")
>>
>>
>>             pair("IP", "${IP}")
>>
>>
>>         )
>>
>>
>>     );
>>
>>
>> };
>>
>>
>>
>>
>>
>> log {
>>
>>
>>     source(s_apache_access_log);
>>
>>
>>     filter(f_apache_access_log);
>>
>>
>>     rewrite(r_apache_access_log);
>>
>>
>>     destination(d_apache_access_log);
>>
>>
>> };
>>
>>
>>
>> but I think something is not ok, I'm not sure this is the right way to do
>> it.
>>
>> This log produces an strange behavior:
>>
>> www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1"
>> 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>> rv:54.0) Gecko/20100101 Firefox/54.0"
>>
>> but this one doesn't
>>
>> cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200
>> 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>> rv:54.0) Gecko/20100101 Firefox/54.0"
>>
>> The behavior is (only for subdomains):
>>
>> DOMAIN: ': www.cifa.li'
>>
>> corret one
>>
>> DOMAIN: 'www.cifa.li'
>>
>> The subdomain seems like it's adding stuff that I didn't (or want) to add.
>>
>>
>> Am I missing something?
>>
>> Thanks in advance.
>>
>>
>> --
>> [ ]'s
>>
>> Filipe Cifali Stangler
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>


-- 
[ ]'s

Filipe Cifali Stangler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170711/26694954/attachment-0001.html>