[syslog-ng] Extract portion of SDATA file name?

[Matt Zagrabelny]

On Tue, Feb 21, 2012 at 11:45 AM, Andika Daud <adaud at adobe.com> wrote:
> Hi,
>
>
>
> I’m a noobie and just about beginning to try out syslog-ng at my company.  I
> have the following configuration on the client:
>
>
>
> source s_apache {
>
>         file("/apps/log/apache/store-dev03/*"
>
>         program_override("apache")
>
>         flags(no-parse)
>
>         );
>
> };
>
> destination d_remote {
>
>     syslog("10.5.76.125" transport("tcp") port(514)
>
> #    template("${MSGONLY}")
>
>     );
>
> };
>
> log {
>
>         source(s_apache);
>
>         destination(d_remote);
>
> };
>
>
>
> And something like this on the server side:
>
> source s_network {
>
>    syslog(ip(0.0.0.0)
>
>         port(514) transport("tcp") );
>
> };
>
> destination d_local {
>
>     file("/apps/log/syslong-ng/$PROGRAM/$HOST/$YEAR$MONTH$DAY.log"
>
>     # ignore the use of template below, I just wanted to prove the server
> side could
>
>     # see the value of the filepath from the client
>
>     template("${.SDATA.file at 18372.4.name} - ${MSGONLY}\n")
>
>     );
>
> };
>
> log {
>
>    source(s_network);
>
>    destination(d_local);
>
> };
>
>
>
> My ("${.SDATA.file at 18372.4.name} value can be something like this:
> ‘/apps/log/apache/store-dev03/access.log’. What I want to be able to do is
> to parse that path.  That is to discard the begining of the path,
> /apps/log/apache.  Get the rest of the value, especially the file name
> (access.log) to be used to construct destination path.
>
>
>
> Is this possible?  Thank you for your kind help.

Hi Andika,

I'm not a syslog-ng expert, but I believe I've done what you are seeking to do.

Here is a chunk of syslog-ng configs that I use for what you describe:

---{begin}---

# Send the following apache log files to bulldog so that people with
shell access can examine
# log messages.
source s_apache_logs {
  file("/var/log/apache2/access.log"
bulldog flags(no-parse));
  file("/var/log/apache2/error.log"
flags(no-parse));
  file("/var/log/apache2/other_vhosts_access.log"
flags(no-parse));
  file("/var/log/apache2/ssl_access.log"
flags(no-parse));
};

destination d_bulldog {
  syslog(
         "bulldog.d.umn.edu"
         transport("tls")
         port(6514)
         tls(
             peer-verify(required-trusted)
             ca_dir('/etc/syslog-ng/ssl/ca.d')
             key_file('/etc/syslog-ng/ssl/server.key')
             cert_file('/etc/syslog-ng/ssl/server.crt')
            )
        );
};

# The funny .SDATA.file at 18372.4.name is for structured data which, I
believe, is part of the (new)
# syslog protocol - RFC5424-formatted (IETF-syslog).

# We need to set the filename value in the SDATA field. $FILE_NAME is
a macro which returns the
# full filename path. ie '/var/log/apache2/access.log'. That will then
get assigned to the
# structured data value, .SDATA.file at 18372.4.name .

rewrite r_setfilename {
  set(
      "$FILE_NAME",
      value(".SDATA.file at 18372.4.name")
     );
};

# After getting the value set for the filename, truncate the directory
portion and only use the
# basename. Use a simple string substitution.

rewrite r_use_basename {
  subst(
        "/var/log/apache2/",
        "",
        value(".SDATA.file at 18372.4.name")
        type("string")
        flags("prefix")
       );
};

log {
  source(s_apache_logs);
  rewrite(r_setfilename);
  rewrite(r_use_basename);
  destination(d_bulldog);
};

---{end}---

Hope that helps,

-mz