[syslog-ng] preserving tags across the network
Balazs Scheidler
bazsi at balabit.hu
Thu Nov 17 21:33:57 CET 2011
On Thu, 2011-11-17 at 08:36 -0600, Matt Zagrabelny wrote:
> 2011/11/12 Balazs Scheidler <bazsi at balabit.hu>:
> > On Fri, 2011-11-11 at 14:01 -0600, Matt Zagrabelny wrote:
> >> On Fri, Nov 11, 2011 at 1:23 PM, Fekete Róbert <frobert at balabit.hu> wrote:
> >> >
> >> > On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe at d.umn.edu> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I am trying to ship a bunch of apache log files across the network and
> >> >> on the syslog-ng server side then break them into their individual
> >> >> files again.
> >> >>
> >> >> I am not sure the best way to do this, but it looks like tags might be helpful.
> >> >
> >> > Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server.
> >> > Like:
> >> > destination d_test {
> >> > file( "/var/log/apache2/${.SDATA.file at 18372.4.name}"
> >> > create_dirs(yes)
> >> > );
> >> > };
> >> >
> >> > For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides/syslog-ng-pe-v4.0-guide-admin-en.html/file-source-and-syslog-destination.html
> >> >
> >> > Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs.
> >>
> >> Hi Róbert,
> >>
> >> Thanks for the hints.
> >>
> >> I am using syslog-ng from Debian Squeeze:
> >>
> >> ii syslog-ng 3.1.3-3
> >> Next generation logging daemon
> >>
> >> I have the following configured now on the server:
> >>
> >> destination d_web2_access {
> >> file(
> >> "/tmp/apache2/${.SDATA.file at 18372.4.name}"
> >> create_dirs(yes)
> >> );
> >> };
> >>
> >> log {
> >> source(s_tls);
> >> destination(d_web2_access);
> >> };
> >>
> >> But all I see is the (newly created) /tmp/apache2 directory.
> >>
> >> I can verify that I am getting apache logs sent over because they are
> >> making it into the "user" facility file for my generic logging. The
> >> relevant config snippet is:
> >>
> >> destination d_remote_clients {
> >> file(
> >> "/var/log/syslog-ng/remote_clients/$HOST_FROM/$YEAR/$MONTH/$DAY/$FACILITY"
> >> owner(root)
> >> group(root)
> >> perm(0644)
> >> dir_perm(0755)
> >> create_dirs(yes)
> >> );
> >> };
> >>
> >> Any other ideas why the SDATA macro is not working?
> >>
> >> Thanks again for the help!
> >
> > I think the automatic addition of the filename into the SDATA is part of
> > the wild-card file source support of the PE version. The code itself is
> > available in the syslog-ng team's public git repository
> > (git.balabit.hu), but I didn't have the time to do a proper review &
> > integration on that code.
> >
> > So this feature is PE only right now, but the code is public, the
> > roadblock is to get enough time on my part, which seems never to
> > happen :( The wild-card file source is a complex piece of code, and I
> > don't want to blindly pull it, since then all the support burden is on
> > me.
> >
> > Anyway, an alternative is to explicitly set the .SDATA field on your
> > client side, as the name of the file is available in the $FILE_NAME
> > macro.
> >
> > This rewrite rule puts the same value in the same SDATA field as the PE
> > does (and the OSE will, once the code gets integrated):
> >
> > rewrite r_setfilename { set("$FILE_NAME" value(".SDATA.file at 18374.4.name"));
> >
> > This should work in 3.1 clients too.
>
> Super. Thanks for the hints, Bazsi. (PS. I think there was a typo in
> the 'file at 18374.4' above. I think it should be 'file at 18372.4'.)
>
> -matt zagrabelny
>
> For anyone grepping the mailing list archive, here are my relevant snippets:
>
> ---{web server syslog-ng configs}---
> # Send the following apache log files to the shell access system so
> that people with shell access can examine
> # log messages.
> source s_apache_logs {
> file("/var/log/apache2/access.log"
> flags(no-parse));
> file("/var/log/apache2/error.log"
> flags(no-parse));
> file("/var/log/apache2/other_vhosts_access.log"
> flags(no-parse));
> file("/var/log/apache2/ssl_access.log"
> flags(no-parse));
> file("/var/log/apache2/suexec.log"
> flags(no-parse));
> };
>
> destination d_shell_access_system {
> syslog(
> "shell-system.domain.com"
> transport("tls")
> port(6514)
> tls(
> peer-verify(required-trusted)
> ca_dir('/etc/syslog-ng/ssl/ca.d')
> key_file('/etc/syslog-ng/ssl/server.key')
> cert_file('/etc/syslog-ng/ssl/server.crt')
> )
> );
> };
>
> # The funny .SDATA.file at 18372.4.name is for structured data which, I
> believe, is part of the (new)
> # syslog protocol - RFC5424-formatted (IETF-syslog).
>
> # We need to set the filename value in the SDATA field. $FILE_NAME is
> a macro which returns the
> # full filename path. ie '/var/log/apache2/access.log'. That will then
> get assigned to the
> # structured data value, .SDATA.file at 18372.4.name .
>
> rewrite r_setfilename {
> set(
> "$FILE_NAME",
> value(".SDATA.file at 18372.4.name")
> );
> };
>
> # After getting the value set for the filename, truncate the directory
> portion and only use the
> # basename. Use a simple string substitution.
>
> rewrite r_use_basename {
> subst(
> "/var/log/apache2/",
> "",
> value(".SDATA.file at 18372.4.name")
> type("string")
> flags("prefix")
> );
> };
note that you can use several rewrite expression in the same rewrite
block, like:
rewrite r_setfilename {
set(
"$FILE_NAME",
value(".SDATA.file at 18372.4.name")
);
subst(
"/var/log/apache2/",
"",
value(".SDATA.file at 18372.4.name")
type("string")
flags("prefix"));
};
This will apply both rewrite expressions one after the other.
>
> log {
> source(s_apache_logs);
> rewrite(r_setfilename);
> rewrite(r_use_basename);
> destination(d_shell_access_system);
> };
> ---{END OF web server syslog-ng configs}---
>
> ---{shell access system syslog-ng configs}---
> source s_tls {
> syslog(
> ip(0.0.0.0)
> port(6514)
> transport("tls")
> tls(
> peer-verify(required-trusted)
> ca_dir('/etc/syslog-ng/ssl/ca.d')
> key_file('/etc/syslog-ng/ssl/server.key')
> cert_file('/etc/syslog-ng/ssl/server.crt')
> )
> max_connections(1000)
> keep_hostname(yes)
> );
> };
>
> destination d_web_log_files {
> file(
> "/usr/local/apache2/log/${.SDATA.file at 18372.4.name}"
> create_dirs(yes)
> );
> };
Please note what happens if the SDATA portion contains '../' or '/..'.
syslog-ng will refuse to open files if that happens.
Also, in the 3.4 tree there's a $(sanitize) template function:
* $(sanitize [opts] $arg1 $arg2 ...)
*
* Options:
* --ctrl-chars or -c Filter control characters (default)
* --no-ctrl-chars or -C Don't filter control characters
* --invalid-chars <set> or -i Set of characters to be translated, default "/"
* --replacement <replace> or -r Single character replacement for invalid chars.
--
Bazsi
More information about the syslog-ng
mailing list