Just to finish this thread,

this may not be the proper way but I got it done by extending the regex to start on ": (.*)"

since I cannot remove the default headers, I have to filter match my message that way (I tried to use message() instead of match() but it did not work)

I only found this after scraping this page:

where I found this part of the docs:https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-source-syslog-chapter.html

no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

It would be good to have an expample on the output inside the syslog so the user knows what to filter or the kind of message he's getting after no-parse.

It would also be wonderful if the docs were linked, so I don't have to search for every term that's not on the same page.

Anyway, thank you for your attention.

On Tue, Jul 11, 2017 at 12:08 PM, Filipe Cifali <cifali.filipe@gmail.com> wrote:

Hi Fabien,

just corrected, but that's not enough info for me, I'm using a combined log and it's not separating the virtualhost (not even setting it)

The apache parser is getting me this kind of log:

{"_apache":{"verb":"HEAD","timestamp":"11/Jul/2017:11:48:48 -0300","response":"200","request":"/h","referrer":"-","rawrequest":"HEAD /h HTTP/1.1","ident":"-","httpversion":"1.1","clientip":"127.0.0.1","bytes":"-","auth":"-","agent":"Monit/5.21.0"}}

coming from this log:

cifa.li 127.0.0.1 - - [11/Jul/2017:11:48:48 -0300] "HEAD /h HTTP/1.1" 200 - "-" "Monit/5.21.0"

On Tue, Jul 11, 2017 at 11:28 AM, Filipe Cifali <cifali.filipe@gmail.com> wrote:
Sorry for the double post, just a heads up on this

if I use source { flags(no-parse) }

the first regex match turns into ": $1"

if I comment flags(no-parse)

the first regex match turns into "$1:"

Since the docs states this:

By default, syslog-ng OSE parses every message as a syslog message. To disable message parsing, use the flags(no-parse) option of the source. To explicitly parse a message as a syslog message, use the syslog parser. For details, see Section 12.1, Parsing syslog messages.

To my understanding, I should use (no-parse) since that message does not come in syslog message format.

On Tue, Jul 11, 2017 at 11:13 AM, Filipe Cifali <cifali.filipe@gmail.com> wrote:
Hi Robert,

I'm on 3.9.1, I have just tried hat example and it returns:

Error parsing affile, Error compiling template, error=Invalid template function reference, missing function name or inbalanced '(', error_pos='24' in /etc/syslog-ng/conf.d/apache.conf at line 54, column 18:
                                                                                                                                        included from /etc/syslog-ng/syslog-ng.conf line 68, column 1

        template("$(format-json .apache.*\n"));
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

Besides the parser itself, the strange part is why the regex returning that extra info at all...

On Tue, Jul 11, 2017 at 10:59 AM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
Hi, in OSE 3.9 and later there is a dedicated apache parser: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/apache-access-log-parser.html

You might want to try it.

HTH,

Robert

On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <cifali.filipe@gmail.com> wrote:
Hi all,

reading the docs I got into this config:

source s_apache_access_log {
    file(
        "/var/logs/apache2/access_log"
        follow-freq(1)
        flags(no-parse)
    );
};

filter f_apache_access_log {
    match(
        '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*) \"-\" (.*)'
        type("pcre")
        flags("store-matches")
    );
};

rewrite r_apache_access_log {
    set("$1", value("DOMAIN") condition(filter(f_apache_access_log)));
    set("$2", value("IP") condition(filter(f_apache_access_log)));
    set("$3", value("HTTP_METHOD") condition(filter(f_apache_access_log)));
    set("$4", value("URI") condition(filter(f_apache_access_log)));
    set("$6", value("HTTP_STATUS") condition(filter(f_apache_access_log)));
    set("$7", value("SIZE") condition(filter(f_apache_access_log)));
    set("$8", value("USER_AGENT") condition(filter(f_apache_access_log)));
};

destination d_apache_access_log {
    mongodb(
        # https://docs.mongodb.com/manual/reference/connection-string/
        persist-name("apache-access-logs")
        uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&socketTimeoutMS=60000&connectTimeoutMS=60000")
        collection("logs")
        retries(3600)
        value-pairs(
            pair("HOST", "${HOST}")
            pair("SERVICE", "APACHE")
            pair("DATE", "${DAY}/${MONTH}/${YEAR}")
            pair("TIME", "${HOUR}:${MIN}")
            pair("MESSAGE", "${MESSAGE}")
            pair("DOMAIN", "${DOMAIN}")
            pair("HTTP_STATUS", "${HTTP_STATUS}")
            pair("HTTP_METHOD", "${HTTP_METHOD}")
            pair("USER_AGENT", "${USER_AGENT}")
            pair("SIZE", "${SIZE}")
            pair("URI", "${URI}")
            pair("IP", "${IP}")
        )
    );
};

log {
    source(s_apache_access_log);
    filter(f_apache_access_log);
    rewrite(r_apache_access_log);
    destination(d_apache_access_log);
};

but I think something is not ok, I'm not sure this is the right way to do it.

This log produces an strange behavior:

www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"

but this one doesn't

cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"

The behavior is (only for subdomains):

DOMAIN: ': www.cifa.li'

corret one

DOMAIN: 'www.cifa.li'

The subdomain seems like it's adding stuff that I didn't (or want) to add.

Am I missing something?

Thanks in advance.

--
[ ]'s

Filipe Cifali Stangler

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

--
[ ]'s

Filipe Cifali Stangler

--
[ ]'s

Filipe Cifali Stangler

--
[ ]'s

Filipe Cifali Stangler

[ ]'s

Filipe Cifali Stangler