Hi Fabien, just corrected, but that's not enough info for me, I'm using a combined log and it's not separating the virtualhost (not even setting it) The apache parser is getting me this kind of log: {"_apache":{"verb":"HEAD","timestamp":"11/Jul/2017:11:48:48 -0300","response":"200","request":"/h","referrer":"-","rawrequest":"HEAD /h HTTP/1.1","ident":"-","httpversion":"1.1","clientip":"127.0.0.1","bytes":"-","auth":"-","agent":"Monit/5.21.0"}} coming from this log: cifa.li 127.0.0.1 - - [11/Jul/2017:11:48:48 -0300] "HEAD /h HTTP/1.1" 200 - "-" "Monit/5.21.0" On Tue, Jul 11, 2017 at 11:28 AM, Filipe Cifali <cifali.filipe@gmail.com> wrote:
Sorry for the double post, just a heads up on this
if I use source { flags(no-parse) }
the first regex match turns into ": $1"
if I comment flags(no-parse)
the first regex match turns into "$1:"
Since the docs states this:
By default, syslog-ng OSE parses every message as a syslog message. To disable message parsing, use the *flags(no-parse)* option of the source. To explicitly parse a message as a syslog message, use the *syslog* parser. For details, see Section 12.1, Parsing syslog messages <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/parser-syslog.html> .
To my understanding, I should use (no-parse) since that message does not come in syslog message format.
On Tue, Jul 11, 2017 at 11:13 AM, Filipe Cifali <cifali.filipe@gmail.com> wrote:
Hi Robert,
I'm on 3.9.1, I have just tried hat example and it returns:
Error parsing affile, Error compiling template, error=Invalid template function reference, missing function name or inbalanced '(', error_pos='24' in /etc/syslog-ng/conf.d/apache.conf at line 54, column 18:
included from /etc/syslog-ng/syslog-ng.conf line 68, column 1
template("$(format-json .apache.*\n")); ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Besides the parser itself, the strange part is why the regex returning that extra info at all...
On Tue, Jul 11, 2017 at 10:59 AM, Fekete, Róbert < robert.fekete@balabit.com> wrote:
Hi, in OSE 3.9 and later there is a dedicated apache parser: https://www.balabit.com/documents/syslog-ng-ose-late st-guides/en/syslog-ng-ose-guide-admin/html/apache-access- log-parser.html
You might want to try it.
HTH,
Robert
On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <cifali.filipe@gmail.com> wrote:
Hi all,
reading the docs I got into this config:
source s_apache_access_log {
file(
"/var/logs/apache2/access_log"
follow-freq(1)
flags(no-parse)
);
};
filter f_apache_access_log {
match(
'(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0- 9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*) \"-\" (.*)' type("pcre")
flags("store-matches")
);
};
rewrite r_apache_access_log {
set("$1", value("DOMAIN") condition(filter(f_apache_acce ss_log)));
set("$2", value("IP") condition(filter(f_apache_acce ss_log)));
set("$3", value("HTTP_METHOD") condition(filter(f_apache_acce ss_log)));
set("$4", value("URI") condition(filter(f_apache_acce ss_log)));
set("$6", value("HTTP_STATUS") condition(filter(f_apache_acce ss_log)));
set("$7", value("SIZE") condition(filter(f_apache_acce ss_log)));
set("$8", value("USER_AGENT") condition(filter(f_apache_acce ss_log)));
};
destination d_apache_access_log {
mongodb(
# https://docs.mongodb.com/manual/reference/connection-string/
persist-name("apache-access-logs")
uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&sock etTimeoutMS=60000&connectTimeoutMS=60000")
collection("logs")
retries(3600)
value-pairs(
pair("HOST", "${HOST}")
pair("SERVICE", "APACHE")
pair("DATE", "${DAY}/${MONTH}/${YEAR}")
pair("TIME", "${HOUR}:${MIN}")
pair("MESSAGE", "${MESSAGE}")
pair("DOMAIN", "${DOMAIN}")
pair("HTTP_STATUS", "${HTTP_STATUS}")
pair("HTTP_METHOD", "${HTTP_METHOD}")
pair("USER_AGENT", "${USER_AGENT}")
pair("SIZE", "${SIZE}")
pair("URI", "${URI}")
pair("IP", "${IP}")
)
);
};
log {
source(s_apache_access_log);
filter(f_apache_access_log);
rewrite(r_apache_access_log);
destination(d_apache_access_log);
};
but I think something is not ok, I'm not sure this is the right way to do it.
This log produces an strange behavior:
www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
but this one doesn't
cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
The behavior is (only for subdomains):
DOMAIN: ': www.cifa.li'
corret one
DOMAIN: 'www.cifa.li'
The subdomain seems like it's adding stuff that I didn't (or want) to add.
Am I missing something?
Thanks in advance.
-- [ ]'s
Filipe Cifali Stangler
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- [ ]'s
Filipe Cifali Stangler
-- [ ]'s
Filipe Cifali Stangler
-- [ ]'s Filipe Cifali Stangler