[syslog-ng] Problem parsing apache logs

7 Jan 2013

      Hello, list.

I've been trying to parse my apache logs in order to separate log messages
into different files based on a column provided by the parser, which in my
case is $APACHE.SERVER_NAME.

It's pretty much the same as the example available on the Administrator
Guide, in section 3.8, p 60.

Looks like I'm close to get it working but there are two problems which I
could not solve:

1- The first field of the log (APACHE.CLIENT_IP) vanishes when it reaches
its final destination and for that reason APACHE.SERVER_NAME, which is the
last column in the parser is empty. That messes up the file creation on the
server. Not mentioning the important log data loss.

2- Syslog-ng server only logs for a few seconds and suddenly stops.
Whenever the client is reloaded, it starts logging for a few seconds again.
Nothing useful for debugging shows up at the server or client internal logs.

Hope you guys can lend me a hand here. Please let me know if you need any
other information.

Here are the meaningful config sections on both the server and the client.

(I think it's worth mentioning that the sources, destinations and the
parser below are not used by any other log paths.)

########### SERVER

options {
        flush_lines (0);
        time_reopen (10);
        log_fifo_size (1000);
        long_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
        owner("root");
        group("adm");
        perm(0640);
};

source s_remote_apache {
        tcp(ip(0.0.0.0) port(51480));
};

destination d_apache {
file("/var/log/httpd/${APACHE.SERVER_NAME:-nouser}.log"
template("MSGONLY\n")); };

parser p_apache {
    csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME",
"APACHE.USER_NAME",
                        "APACHE.TIMESTAMP", "APACHE.REQUEST_URL",
"APACHE.REQUEST_STATUS",
                        "APACHE.CONTENT_LENGTH", "APACHE.REFERER",
"APACHE.USER_AGENT",
                        "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
                flags(escape-double-char,strip-whitespace)
                delimiters(" ")
                quote-pairs('""[]')
                );
};

log { source(s_remote_apache); parser(p_apache); destination(d_apache); };

############ CLIENT

options {
        sync (0);
        time_reopen (10);
        log_fifo_size (1000);
        long_hostnames (off);
        use_dns (yes);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
};

source s_apache {
        file ("/var/log/httpd/access_log" flags(no-parse));
};

destination d_apache_remote {
        tcp("myhost.example.com" port(51480));
};

log { source(s_apache); destination(d_apache_remote); };

Luis Alen
www.izap.com.br
Ligue com tarifa local de todo o Brasil 4020.3000

[syslog-ng] Problem parsing apache logs

Luis Fernando Alen