[syslog-ng] New user trying to filter / rewrite apache logs

Filipe Cifali cifali.filipe at gmail.com
Tue Jul 11 15:08:15 UTC 2017 

Hi Fabien,

just corrected, but that's not enough info for me, I'm using a combined log
and it's not separating the virtualhost (not even setting it)

The apache parser is getting me this kind of log:

{"_apache":{"verb":"HEAD","timestamp":"11/Jul/2017:11:48:48
-0300","response":"200","request":"/h","referrer":"-","rawrequest":"HEAD /h
HTTP/1.1","ident":"-","httpversion":"1.1","clientip":"127.0.0.1","bytes":"-","auth":"-","agent":"Monit/5.21.0"}}

coming from this log:

cifa.li 127.0.0.1 - - [11/Jul/2017:11:48:48 -0300] "HEAD /h HTTP/1.1" 200 -
"-" "Monit/5.21.0"




On Tue, Jul 11, 2017 at 11:28 AM, Filipe Cifali <cifali.filipe at gmail.com>
wrote:

> Sorry for the double post, just a heads up on this
>
> if I use source { flags(no-parse) }
>
> the first regex match turns into ": $1"
>
> if I comment flags(no-parse)
>
> the first regex match turns into "$1:"
>
> Since the docs states this:
>
> By default, syslog-ng OSE parses every message as a syslog message. To
> disable message parsing, use the *flags(no-parse)* option of the source.
> To explicitly parse a message as a syslog message, use the *syslog*
> parser. For details, see Section 12.1, Parsing syslog messages
> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/parser-syslog.html>
> .
>
> To my understanding, I should use (no-parse) since that message does not
> come in syslog message format.
>
> On Tue, Jul 11, 2017 at 11:13 AM, Filipe Cifali <cifali.filipe at gmail.com>
> wrote:
>
>> Hi Robert,
>>
>> I'm on 3.9.1, I have just tried hat example and it returns:
>>
>> Error parsing affile, Error compiling template, error=Invalid template
>> function reference, missing function name or inbalanced '(', error_pos='24'
>> in /etc/syslog-ng/conf.d/apache.conf at line 54, column 18:
>>
>>
>> included from /etc/syslog-ng/syslog-ng.conf line 68, column 1
>>
>>         template("$(format-json .apache.*\n"));
>>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> Besides the parser itself, the strange part is why the regex returning
>> that extra info at all...
>>
>>
>> On Tue, Jul 11, 2017 at 10:59 AM, Fekete, Róbert <
>> robert.fekete at balabit.com> wrote:
>>
>>> Hi, in OSE 3.9 and later there is a dedicated apache parser:
>>> https://www.balabit.com/documents/syslog-ng-ose-late
>>> st-guides/en/syslog-ng-ose-guide-admin/html/apache-access-
>>> log-parser.html
>>>
>>> You might want to try it.
>>>
>>> HTH,
>>>
>>> Robert
>>>
>>> On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <cifali.filipe at gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> reading the docs I got into this config:
>>>>
>>>> source s_apache_access_log {
>>>>
>>>>
>>>>     file(
>>>>
>>>>
>>>>         "/var/logs/apache2/access_log"
>>>>
>>>>
>>>>         follow-freq(1)
>>>>
>>>>
>>>>         flags(no-parse)
>>>>
>>>>
>>>>     );
>>>>
>>>>
>>>> };
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> filter f_apache_access_log {
>>>>
>>>>
>>>>     match(
>>>>
>>>>
>>>>         '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-
>>>> 9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*)
>>>> \"-\" (.*)'
>>>>         type("pcre")
>>>>
>>>>
>>>>         flags("store-matches")
>>>>
>>>>
>>>>     );
>>>>
>>>>
>>>> };
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> rewrite r_apache_access_log {
>>>>
>>>>
>>>>     set("$1", value("DOMAIN") condition(filter(f_apache_acce
>>>> ss_log)));
>>>>
>>>>     set("$2", value("IP") condition(filter(f_apache_acce
>>>> ss_log)));
>>>>
>>>>     set("$3", value("HTTP_METHOD") condition(filter(f_apache_acce
>>>> ss_log)));
>>>>
>>>>     set("$4", value("URI") condition(filter(f_apache_acce
>>>> ss_log)));
>>>>
>>>>     set("$6", value("HTTP_STATUS") condition(filter(f_apache_acce
>>>> ss_log)));
>>>>
>>>>     set("$7", value("SIZE") condition(filter(f_apache_acce
>>>> ss_log)));
>>>>
>>>>     set("$8", value("USER_AGENT") condition(filter(f_apache_acce
>>>> ss_log)));
>>>>
>>>> };
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> destination d_apache_access_log {
>>>>
>>>>
>>>>     mongodb(
>>>>
>>>>
>>>>         # https://docs.mongodb.com/manual/reference/connection-string/
>>>>
>>>>
>>>>         persist-name("apache-access-logs")
>>>>
>>>>
>>>>         uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&sock
>>>> etTimeoutMS=60000&connectTimeoutMS=60000")
>>>>
>>>>         collection("logs")
>>>>
>>>>
>>>>         retries(3600)
>>>>
>>>>
>>>>         value-pairs(
>>>>
>>>>
>>>>             pair("HOST", "${HOST}")
>>>>
>>>>
>>>>             pair("SERVICE", "APACHE")
>>>>
>>>>
>>>>             pair("DATE", "${DAY}/${MONTH}/${YEAR}")
>>>>
>>>>
>>>>             pair("TIME", "${HOUR}:${MIN}")
>>>>
>>>>
>>>>             pair("MESSAGE", "${MESSAGE}")
>>>>
>>>>
>>>>             pair("DOMAIN", "${DOMAIN}")
>>>>
>>>>
>>>>             pair("HTTP_STATUS", "${HTTP_STATUS}")
>>>>
>>>>
>>>>             pair("HTTP_METHOD", "${HTTP_METHOD}")
>>>>
>>>>
>>>>             pair("USER_AGENT", "${USER_AGENT}")
>>>>
>>>>
>>>>             pair("SIZE", "${SIZE}")
>>>>
>>>>
>>>>             pair("URI", "${URI}")
>>>>
>>>>
>>>>             pair("IP", "${IP}")
>>>>
>>>>
>>>>         )
>>>>
>>>>
>>>>     );
>>>>
>>>>
>>>> };
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> log {
>>>>
>>>>
>>>>     source(s_apache_access_log);
>>>>
>>>>
>>>>     filter(f_apache_access_log);
>>>>
>>>>
>>>>     rewrite(r_apache_access_log);
>>>>
>>>>
>>>>     destination(d_apache_access_log);
>>>>
>>>>
>>>> };
>>>>
>>>>
>>>>
>>>> but I think something is not ok, I'm not sure this is the right way to
>>>> do it.
>>>>
>>>> This log produces an strange behavior:
>>>>
>>>> www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET /
>>>> HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux
>>>> x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
>>>>
>>>> but this one doesn't
>>>>
>>>> cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1"
>>>> 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>>>> rv:54.0) Gecko/20100101 Firefox/54.0"
>>>>
>>>> The behavior is (only for subdomains):
>>>>
>>>> DOMAIN: ': www.cifa.li'
>>>>
>>>> corret one
>>>>
>>>> DOMAIN: 'www.cifa.li'
>>>>
>>>> The subdomain seems like it's adding stuff that I didn't (or want) to
>>>> add.
>>>>
>>>>
>>>> Am I missing something?
>>>>
>>>> Thanks in advance.
>>>>
>>>>
>>>> --
>>>> [ ]'s
>>>>
>>>> Filipe Cifali Stangler
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> --
>> [ ]'s
>>
>> Filipe Cifali Stangler
>>
>
>
>
> --
> [ ]'s
>
> Filipe Cifali Stangler
>



-- 
[ ]'s

Filipe Cifali Stangler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170711/e8a3eadb/attachment-0001.html>

More information about the syslog-ng mailing list