[syslog-ng] New user trying to filter / rewrite apache logs
Filipe Cifali
cifali.filipe at gmail.com
Tue Jul 11 16:36:27 UTC 2017
Just to finish this thread,
this may not be the proper way but I got it done by extending the regex to
start on ": (.*)"
since I cannot remove the default headers, I have to filter match my
message that way (I tried to use message() instead of match() but it did
not work)
I only found this after scraping this page:
where I found this part of the docs:
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-source-syslog-chapter.html
*no-parse*: By default, syslog-ng OSE parses incoming messages as syslog
messages. The *no-parse* flag completely disables syslog message parsing
and processes the complete line as the message part of a syslog message. *The
syslog-ng OSE application will generate a new syslog header (timestamp,
host, and so on) automatically and put the entire incoming message into the
MESSAGE part of the syslog message (available using the ${MESSAGE} macro).*
This flag is useful for parsing messages not complying to the syslog format.
It would be good to have an expample on the output inside the syslog so the
user knows what to filter or the kind of message he's getting after
no-parse.
It would also be wonderful if the docs were linked, so I don't have to
search for every term that's not on the same page.
Anyway, thank you for your attention.
On Tue, Jul 11, 2017 at 12:08 PM, Filipe Cifali <cifali.filipe at gmail.com>
wrote:
> Hi Fabien,
>
> just corrected, but that's not enough info for me, I'm using a combined
> log and it's not separating the virtualhost (not even setting it)
>
> The apache parser is getting me this kind of log:
>
> {"_apache":{"verb":"HEAD","timestamp":"11/Jul/2017:11:48:48
> -0300","response":"200","request":"/h","referrer":"-","rawrequest":"HEAD
> /h HTTP/1.1","ident":"-","httpversion":"1.1","clientip":
> "127.0.0.1","bytes":"-","auth":"-","agent":"Monit/5.21.0"}}
>
> coming from this log:
>
> cifa.li 127.0.0.1 - - [11/Jul/2017:11:48:48 -0300] "HEAD /h HTTP/1.1" 200
> - "-" "Monit/5.21.0"
>
>
>
>
> On Tue, Jul 11, 2017 at 11:28 AM, Filipe Cifali <cifali.filipe at gmail.com>
> wrote:
>
>> Sorry for the double post, just a heads up on this
>>
>> if I use source { flags(no-parse) }
>>
>> the first regex match turns into ": $1"
>>
>> if I comment flags(no-parse)
>>
>> the first regex match turns into "$1:"
>>
>> Since the docs states this:
>>
>> By default, syslog-ng OSE parses every message as a syslog message. To
>> disable message parsing, use the *flags(no-parse)* option of the source.
>> To explicitly parse a message as a syslog message, use the *syslog*
>> parser. For details, see Section 12.1, Parsing syslog messages
>> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/parser-syslog.html>
>> .
>>
>> To my understanding, I should use (no-parse) since that message does not
>> come in syslog message format.
>>
>> On Tue, Jul 11, 2017 at 11:13 AM, Filipe Cifali <cifali.filipe at gmail.com>
>> wrote:
>>
>>> Hi Robert,
>>>
>>> I'm on 3.9.1, I have just tried hat example and it returns:
>>>
>>> Error parsing affile, Error compiling template, error=Invalid template
>>> function reference, missing function name or inbalanced '(', error_pos='24'
>>> in /etc/syslog-ng/conf.d/apache.conf at line 54, column 18:
>>>
>>>
>>> included from /etc/syslog-ng/syslog-ng.conf line 68, column 1
>>>
>>> template("$(format-json .apache.*\n"));
>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>
>>> Besides the parser itself, the strange part is why the regex returning
>>> that extra info at all...
>>>
>>>
>>> On Tue, Jul 11, 2017 at 10:59 AM, Fekete, Róbert <
>>> robert.fekete at balabit.com> wrote:
>>>
>>>> Hi, in OSE 3.9 and later there is a dedicated apache parser:
>>>> https://www.balabit.com/documents/syslog-ng-ose-late
>>>> st-guides/en/syslog-ng-ose-guide-admin/html/apache-access-lo
>>>> g-parser.html
>>>>
>>>> You might want to try it.
>>>>
>>>> HTH,
>>>>
>>>> Robert
>>>>
>>>> On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <cifali.filipe at gmail.com
>>>> > wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> reading the docs I got into this config:
>>>>>
>>>>> source s_apache_access_log {
>>>>>
>>>>>
>>>>> file(
>>>>>
>>>>>
>>>>> "/var/logs/apache2/access_log"
>>>>>
>>>>>
>>>>> follow-freq(1)
>>>>>
>>>>>
>>>>> flags(no-parse)
>>>>>
>>>>>
>>>>> );
>>>>>
>>>>>
>>>>> };
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> filter f_apache_access_log {
>>>>>
>>>>>
>>>>> match(
>>>>>
>>>>>
>>>>> '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-
>>>>> 9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*)
>>>>> \"-\" (.*)'
>>>>> type("pcre")
>>>>>
>>>>>
>>>>> flags("store-matches")
>>>>>
>>>>>
>>>>> );
>>>>>
>>>>>
>>>>> };
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> rewrite r_apache_access_log {
>>>>>
>>>>>
>>>>> set("$1", value("DOMAIN") condition(filter(f_apache_acce
>>>>> ss_log)));
>>>>>
>>>>> set("$2", value("IP") condition(filter(f_apache_acce
>>>>> ss_log)));
>>>>>
>>>>> set("$3", value("HTTP_METHOD") condition(filter(f_apache_acce
>>>>> ss_log)));
>>>>>
>>>>> set("$4", value("URI") condition(filter(f_apache_acce
>>>>> ss_log)));
>>>>>
>>>>> set("$6", value("HTTP_STATUS") condition(filter(f_apache_acce
>>>>> ss_log)));
>>>>>
>>>>> set("$7", value("SIZE") condition(filter(f_apache_acce
>>>>> ss_log)));
>>>>>
>>>>> set("$8", value("USER_AGENT") condition(filter(f_apache_acce
>>>>> ss_log)));
>>>>>
>>>>> };
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> destination d_apache_access_log {
>>>>>
>>>>>
>>>>> mongodb(
>>>>>
>>>>>
>>>>> # https://docs.mongodb.com/manual/reference/connection-string/
>>>>>
>>>>>
>>>>> persist-name("apache-access-logs")
>>>>>
>>>>>
>>>>> uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&sock
>>>>> etTimeoutMS=60000&connectTimeoutMS=60000")
>>>>>
>>>>> collection("logs")
>>>>>
>>>>>
>>>>> retries(3600)
>>>>>
>>>>>
>>>>> value-pairs(
>>>>>
>>>>>
>>>>> pair("HOST", "${HOST}")
>>>>>
>>>>>
>>>>> pair("SERVICE", "APACHE")
>>>>>
>>>>>
>>>>> pair("DATE", "${DAY}/${MONTH}/${YEAR}")
>>>>>
>>>>>
>>>>> pair("TIME", "${HOUR}:${MIN}")
>>>>>
>>>>>
>>>>> pair("MESSAGE", "${MESSAGE}")
>>>>>
>>>>>
>>>>> pair("DOMAIN", "${DOMAIN}")
>>>>>
>>>>>
>>>>> pair("HTTP_STATUS", "${HTTP_STATUS}")
>>>>>
>>>>>
>>>>> pair("HTTP_METHOD", "${HTTP_METHOD}")
>>>>>
>>>>>
>>>>> pair("USER_AGENT", "${USER_AGENT}")
>>>>>
>>>>>
>>>>> pair("SIZE", "${SIZE}")
>>>>>
>>>>>
>>>>> pair("URI", "${URI}")
>>>>>
>>>>>
>>>>> pair("IP", "${IP}")
>>>>>
>>>>>
>>>>> )
>>>>>
>>>>>
>>>>> );
>>>>>
>>>>>
>>>>> };
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> log {
>>>>>
>>>>>
>>>>> source(s_apache_access_log);
>>>>>
>>>>>
>>>>> filter(f_apache_access_log);
>>>>>
>>>>>
>>>>> rewrite(r_apache_access_log);
>>>>>
>>>>>
>>>>> destination(d_apache_access_log);
>>>>>
>>>>>
>>>>> };
>>>>>
>>>>>
>>>>>
>>>>> but I think something is not ok, I'm not sure this is the right way to
>>>>> do it.
>>>>>
>>>>> This log produces an strange behavior:
>>>>>
>>>>> www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET /
>>>>> HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu;
>>>>> Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
>>>>>
>>>>> but this one doesn't
>>>>>
>>>>> cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1"
>>>>> 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
>>>>> rv:54.0) Gecko/20100101 Firefox/54.0"
>>>>>
>>>>> The behavior is (only for subdomains):
>>>>>
>>>>> DOMAIN: ': www.cifa.li'
>>>>>
>>>>> corret one
>>>>>
>>>>> DOMAIN: 'www.cifa.li'
>>>>>
>>>>> The subdomain seems like it's adding stuff that I didn't (or want) to
>>>>> add.
>>>>>
>>>>>
>>>>> Am I missing something?
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>>
>>>>> --
>>>>> [ ]'s
>>>>>
>>>>> Filipe Cifali Stangler
>>>>>
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support
>>>>> /documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> [ ]'s
>>>
>>> Filipe Cifali Stangler
>>>
>>
>>
>>
>> --
>> [ ]'s
>>
>> Filipe Cifali Stangler
>>
>
>
>
> --
> [ ]'s
>
> Filipe Cifali Stangler
>
--
[ ]'s
Filipe Cifali Stangler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170711/96283f70/attachment-0001.html>
More information about the syslog-ng
mailing list