[syslog-ng] Extract portion of SDATA file name?

[Andika Daud]

It works like a charm.  Thank you Matt!

Andika Daud  |  Sr. Web Technologist  |  Adobe  |  p. 408.536.4713  | adaud at adobe.com


-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Matt Zagrabelny
Sent: Tuesday, February 21, 2012 10:18 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Extract portion of SDATA file name?

On Tue, Feb 21, 2012 at 11:45 AM, Andika Daud <adaud at adobe.com> wrote:
> Hi,
>
>
>
> I'm a noobie and just about beginning to try out syslog-ng at my 
> company.  I have the following configuration on the client:
>
>
>
> source s_apache {
>
>         file("/apps/log/apache/store-dev03/*"
>
>         program_override("apache")
>
>         flags(no-parse)
>
>         );
>
> };
>
> destination d_remote {
>
>     syslog("10.5.76.125" transport("tcp") port(514)
>
> #    template("${MSGONLY}")
>
>     );
>
> };
>
> log {
>
>         source(s_apache);
>
>         destination(d_remote);
>
> };
>
>
>
> And something like this on the server side:
>
> source s_network {
>
>    syslog(ip(0.0.0.0)
>
>         port(514) transport("tcp") );
>
> };
>
> destination d_local {
>
>     file("/apps/log/syslong-ng/$PROGRAM/$HOST/$YEAR$MONTH$DAY.log"
>
>     # ignore the use of template below, I just wanted to prove the 
> server side could
>
>     # see the value of the filepath from the client
>
>     template("${.SDATA.file at 18372.4.name} - ${MSGONLY}\n")
>
>     );
>
> };
>
> log {
>
>    source(s_network);
>
>    destination(d_local);
>
> };
>
>
>
> My ("${.SDATA.file at 18372.4.name} value can be something like this:
> '/apps/log/apache/store-dev03/access.log'. What I want to be able to 
> do is to parse that path.  That is to discard the begining of the 
> path, /apps/log/apache.  Get the rest of the value, especially the 
> file name
> (access.log) to be used to construct destination path.
>
>
>
> Is this possible?  Thank you for your kind help.

Hi Andika,

I'm not a syslog-ng expert, but I believe I've done what you are seeking to do.

Here is a chunk of syslog-ng configs that I use for what you describe:

---{begin}---

# Send the following apache log files to bulldog so that people with shell access can examine # log messages.
source s_apache_logs {
  file("/var/log/apache2/access.log"
bulldog flags(no-parse));
  file("/var/log/apache2/error.log"
flags(no-parse));
  file("/var/log/apache2/other_vhosts_access.log"
flags(no-parse));
  file("/var/log/apache2/ssl_access.log"
flags(no-parse));
};

destination d_bulldog {
  syslog(
         "bulldog.d.umn.edu"
         transport("tls")
         port(6514)
         tls(
             peer-verify(required-trusted)
             ca_dir('/etc/syslog-ng/ssl/ca.d')
             key_file('/etc/syslog-ng/ssl/server.key')
             cert_file('/etc/syslog-ng/ssl/server.crt')
            )
        );
};

# The funny .SDATA.file at 18372.4.name is for structured data which, I believe, is part of the (new) # syslog protocol - RFC5424-formatted (IETF-syslog).

# We need to set the filename value in the SDATA field. $FILE_NAME is a macro which returns the # full filename path. ie '/var/log/apache2/access.log'. That will then get assigned to the # structured data value, .SDATA.file at 18372.4.name .

rewrite r_setfilename {
  set(
      "$FILE_NAME",
      value(".SDATA.file at 18372.4.name")
     );
};

# After getting the value set for the filename, truncate the directory portion and only use the # basename. Use a simple string substitution.

rewrite r_use_basename {
  subst(
        "/var/log/apache2/",
        "",
        value(".SDATA.file at 18372.4.name")
        type("string")
        flags("prefix")
       );
};

log {
  source(s_apache_logs);
  rewrite(r_setfilename);
  rewrite(r_use_basename);
  destination(d_bulldog);
};

---{end}---

Hope that helps,

-mz
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq