[syslog-ng] New user trying to filter / rewrite apache logs
Filipe Cifali
cifali.filipe at gmail.com
Tue Jul 11 14:28:10 UTC 2017
Sorry for the double post, just a heads up on this
if I use source { flags(no-parse) }
the first regex match turns into ": $1"
if I comment flags(no-parse)
the first regex match turns into "$1:"
Since the docs states this:
By default, syslog-ng OSE parses every message as a syslog message. To
disable message parsing, use the *flags(no-parse)* option of the source. To
explicitly parse a message as a syslog message, use the *syslog* parser.
For details, see Section 12.1, Parsing syslog messages
<https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/parser-syslog.html>
.
To my understanding, I should use (no-parse) since that message does not
come in syslog message format.
On Tue, Jul 11, 2017 at 11:13 AM, Filipe Cifali <cifali.filipe at gmail.com>
wrote:
> Hi Robert,
>
> I'm on 3.9.1, I have just tried hat example and it returns:
>
> Error parsing affile, Error compiling template, error=Invalid template
> function reference, missing function name or inbalanced '(', error_pos='24'
> in /etc/syslog-ng/conf.d/apache.conf at line 54, column 18:
>
>
> included from /etc/syslog-ng/syslog-ng.conf line 68, column 1
>
> template("$(format-json .apache.*\n"));
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Besides the parser itself, the strange part is why the regex returning
> that extra info at all...
>
>
> On Tue, Jul 11, 2017 at 10:59 AM, Fekete, Róbert <
> robert.fekete at balabit.com> wrote:
>
>> Hi, in OSE 3.9 and later there is a dedicated apache parser:
>> https://www.balabit.com/documents/syslog-ng-ose-late
>> st-guides/en/syslog-ng-ose-guide-admin/html/apache-access-log-parser.html
>>
>> You might want to try it.
>>
>> HTH,
>>
>> Robert
>>
>> On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <cifali.filipe at gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> reading the docs I got into this config:
>>>
>>> source s_apache_access_log {
>>>
>>>
>>> file(
>>>
>>>
>>> "/var/logs/apache2/access_log"
>>>
>>>
>>> follow-freq(1)
>>>
>>>
>>> flags(no-parse)
>>>
>>>
>>> );
>>>
>>>
>>> };
>>>
>>>
>>>
>>>
>>>
>>> filter f_apache_access_log {
>>>
>>>
>>> match(
>>>
>>>
>>> '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-
>>> 9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*)
>>> \"-\" (.*)'
>>> type("pcre")
>>>
>>>
>>> flags("store-matches")
>>>
>>>
>>> );
>>>
>>>
>>> };
>>>
>>>
>>>
>>>
>>>
>>> rewrite r_apache_access_log {
>>>
>>>
>>> set("$1", value("DOMAIN") condition(filter(f_apache_acce
>>> ss_log)));
>>>
>>> set("$2", value("IP") condition(filter(f_apache_acce
>>> ss_log)));
>>>
>>> set("$3", value("HTTP_METHOD") condition(filter(f_apache_acce
>>> ss_log)));
>>>
>>> set("$4", value("URI") condition(filter(f_apache_acce
>>> ss_log)));
>>>
>>> set("$6", value("HTTP_STATUS") condition(filter(f_apache_acce
>>> ss_log)));
>>>
>>> set("$7", value("SIZE") condition(filter(f_apache_acce
>>> ss_log)));
>>>
>>> set("$8", value("USER_AGENT") condition(filter(f_apache_acce
>>> ss_log)));
>>>
>>> };
>>>
>>>
>>>
>>>
>>>
>>> destination d_apache_access_log {
>>>
>>>
>>> mongodb(
>>>
>>>
>>> # https://docs.mongodb.com/manual/reference/connection-string/
>>>
>>>
>>> persist-name("apache-access-logs")
>>>
>>>
>>> uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&sock
>>> etTimeoutMS=60000&connectTimeoutMS=60000")
>>>
>>> collection("logs")
>>>
>>>
>>> retries(3600)
>>>
>>>
>>> value-pairs(
>>>
>>>
>>> pair("HOST", "${HOST}")
>>>
>>>
>>> pair("SERVICE", "APACHE")
>>>
>>>
>>> pair("DATE", "${DAY}/${MONTH}/${YEAR}")
>>>
>>>
>>> pair("TIME", "${HOUR}:${MIN}")
>>>
>>>
>>> pair("MESSAGE", "${MESSAGE}")
>>>
>>>
>>> pair("DOMAIN", "${DOMAIN}")
>>>
>>>
>>> pair("HTTP_STATUS", "${HTTP_STATUS}")
>>>
>>>
>>> pair("HTTP_METHOD", "${HTTP_METHOD}")
>>>
>>>
>>> pair("USER_AGENT", "${USER_AGENT}")
>>>
>>>
>>> pair("SIZE", "${SIZE}")
>>>
>>>
>>> pair("URI", "${URI}")
>>>
>>>
>>> pair("IP", "${IP}")
>>>
>>>
>>> )
>>>
>>>
>>> );
>>>
>>>
>>> };
>>>
>>>
>>>
>>>
>>>
>>> log {
>>>
>>>
>>> source(s_apache_access_log);
>>>
>>>
>>> filter(f_apache_access_log);
>>>
>>>
>>> rewrite(r_apache_access_log);
>>>
>>>
>>> destination(d_apache_access_log);
>>>
>>>
>>> };
>>>
>>>
>>>
>>> but I think something is not ok, I'm not sure this is the right way to
>>> do it.
>>>
>>> This log produces an strange behavior:
>>>
>>> www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1"
>>> 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>>> rv:54.0) Gecko/20100101 Firefox/54.0"
>>>
>>> but this one doesn't
>>>
>>> cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200
>>> 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>>> rv:54.0) Gecko/20100101 Firefox/54.0"
>>>
>>> The behavior is (only for subdomains):
>>>
>>> DOMAIN: ': www.cifa.li'
>>>
>>> corret one
>>>
>>> DOMAIN: 'www.cifa.li'
>>>
>>> The subdomain seems like it's adding stuff that I didn't (or want) to
>>> add.
>>>
>>>
>>> Am I missing something?
>>>
>>> Thanks in advance.
>>>
>>>
>>> --
>>> [ ]'s
>>>
>>> Filipe Cifali Stangler
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
> [ ]'s
>
> Filipe Cifali Stangler
>
--
[ ]'s
Filipe Cifali Stangler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170711/2eebabe8/attachment-0001.html>
More information about the syslog-ng
mailing list