Re: [syslog-ng] Extract portion of SDATA file name?

21 Feb 2012

      It works like a charm.  Thank you Matt!

Andika Daud  |  Sr. Web Technologist  |  Adobe  |  p. 408.536.4713  | adaud@adobe.com

-----Original Message-----
From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matt Zagrabelny
Sent: Tuesday, February 21, 2012 10:18 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Extract portion of SDATA file name?

On Tue, Feb 21, 2012 at 11:45 AM, Andika Daud <adaud@adobe.com> wrote:
...
Hi,
I'm a noobie and just about beginning to try out syslog-ng at my 
company.  I have the following configuration on the client:
source s_apache {
        file("/apps/log/apache/store-dev03/*"
        program_override("apache")
        flags(no-parse)
        );
};
destination d_remote {
    syslog("10.5.76.125" transport("tcp") port(514)
#    template("${MSGONLY}")
    );
};
log {
        source(s_apache);
        destination(d_remote);
};
And something like this on the server side:
source s_network {
   syslog(ip(0.0.0.0)
        port(514) transport("tcp") );
};
destination d_local {
    file("/apps/log/syslong-ng/$PROGRAM/$HOST/$YEAR$MONTH$DAY.log"
    # ignore the use of template below, I just wanted to prove the 
server side could
    # see the value of the filepath from the client
    template("${.SDATA.file@18372.4.name} - ${MSGONLY}\n")
    );
};
log {
   source(s_network);
   destination(d_local);
};
My ("${.SDATA.file@18372.4.name} value can be something like this:
'/apps/log/apache/store-dev03/access.log'. What I want to be able to 
do is to parse that path.  That is to discard the begining of the 
path, /apps/log/apache.  Get the rest of the value, especially the 
file name
(access.log) to be used to construct destination path.
Is this possible?  Thank you for your kind help.
Hi Andika,

I'm not a syslog-ng expert, but I believe I've done what you are seeking to do.

Here is a chunk of syslog-ng configs that I use for what you describe:

---{begin}---

# Send the following apache log files to bulldog so that people with shell access can examine # log messages.
source s_apache_logs {
  file("/var/log/apache2/access.log"
bulldog flags(no-parse));
  file("/var/log/apache2/error.log"
flags(no-parse));
  file("/var/log/apache2/other_vhosts_access.log"
flags(no-parse));
  file("/var/log/apache2/ssl_access.log"
flags(no-parse));
};

destination d_bulldog {
  syslog(
         "bulldog.d.umn.edu"
         transport("tls")
         port(6514)
         tls(
             peer-verify(required-trusted)
             ca_dir('/etc/syslog-ng/ssl/ca.d')
             key_file('/etc/syslog-ng/ssl/server.key')
             cert_file('/etc/syslog-ng/ssl/server.crt')
            )
        );
};

# The funny .SDATA.file@18372.4.name is for structured data which, I believe, is part of the (new) # syslog protocol - RFC5424-formatted (IETF-syslog).

# We need to set the filename value in the SDATA field. $FILE_NAME is a macro which returns the # full filename path. ie '/var/log/apache2/access.log'. That will then get assigned to the # structured data value, .SDATA.file@18372.4.name .

rewrite r_setfilename {
  set(
      "$FILE_NAME",
      value(".SDATA.file@18372.4.name")
     );
};

# After getting the value set for the filename, truncate the directory portion and only use the # basename. Use a simple string substitution.

rewrite r_use_basename {
  subst(
        "/var/log/apache2/",
        "",
        value(".SDATA.file@18372.4.name")
        type("string")
        flags("prefix")
       );
};

log {
  source(s_apache_logs);
  rewrite(r_setfilename);
  rewrite(r_use_basename);
  destination(d_bulldog);
};

---{end}---

Hope that helps,

-mz
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq