[syslog-ng] preserving tags across the network
Matt Zagrabelny
mzagrabe at d.umn.edu
Thu Nov 17 15:36:17 CET 2011
2011/11/12 Balazs Scheidler <bazsi at balabit.hu>:
> On Fri, 2011-11-11 at 14:01 -0600, Matt Zagrabelny wrote:
>> On Fri, Nov 11, 2011 at 1:23 PM, Fekete Róbert <frobert at balabit.hu> wrote:
>> >
>> > On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe at d.umn.edu> wrote:
>> >
>> >> Hi,
>> >>
>> >> I am trying to ship a bunch of apache log files across the network and
>> >> on the syslog-ng server side then break them into their individual
>> >> files again.
>> >>
>> >> I am not sure the best way to do this, but it looks like tags might be helpful.
>> >
>> > Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server.
>> > Like:
>> > destination d_test {
>> > file( "/var/log/apache2/${.SDATA.file at 18372.4.name}"
>> > create_dirs(yes)
>> > );
>> > };
>> >
>> > For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides/syslog-ng-pe-v4.0-guide-admin-en.html/file-source-and-syslog-destination.html
>> >
>> > Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs.
>>
>> Hi Róbert,
>>
>> Thanks for the hints.
>>
>> I am using syslog-ng from Debian Squeeze:
>>
>> ii syslog-ng 3.1.3-3
>> Next generation logging daemon
>>
>> I have the following configured now on the server:
>>
>> destination d_web2_access {
>> file(
>> "/tmp/apache2/${.SDATA.file at 18372.4.name}"
>> create_dirs(yes)
>> );
>> };
>>
>> log {
>> source(s_tls);
>> destination(d_web2_access);
>> };
>>
>> But all I see is the (newly created) /tmp/apache2 directory.
>>
>> I can verify that I am getting apache logs sent over because they are
>> making it into the "user" facility file for my generic logging. The
>> relevant config snippet is:
>>
>> destination d_remote_clients {
>> file(
>> "/var/log/syslog-ng/remote_clients/$HOST_FROM/$YEAR/$MONTH/$DAY/$FACILITY"
>> owner(root)
>> group(root)
>> perm(0644)
>> dir_perm(0755)
>> create_dirs(yes)
>> );
>> };
>>
>> Any other ideas why the SDATA macro is not working?
>>
>> Thanks again for the help!
>
> I think the automatic addition of the filename into the SDATA is part of
> the wild-card file source support of the PE version. The code itself is
> available in the syslog-ng team's public git repository
> (git.balabit.hu), but I didn't have the time to do a proper review &
> integration on that code.
>
> So this feature is PE only right now, but the code is public, the
> roadblock is to get enough time on my part, which seems never to
> happen :( The wild-card file source is a complex piece of code, and I
> don't want to blindly pull it, since then all the support burden is on
> me.
>
> Anyway, an alternative is to explicitly set the .SDATA field on your
> client side, as the name of the file is available in the $FILE_NAME
> macro.
>
> This rewrite rule puts the same value in the same SDATA field as the PE
> does (and the OSE will, once the code gets integrated):
>
> rewrite r_setfilename { set("$FILE_NAME" value(".SDATA.file at 18374.4.name"));
>
> This should work in 3.1 clients too.
Super. Thanks for the hints, Bazsi. (PS. I think there was a typo in
the 'file at 18374.4' above. I think it should be 'file at 18372.4'.)
-matt zagrabelny
For anyone grepping the mailing list archive, here are my relevant snippets:
---{web server syslog-ng configs}---
# Send the following apache log files to the shell access system so
that people with shell access can examine
# log messages.
source s_apache_logs {
file("/var/log/apache2/access.log"
flags(no-parse));
file("/var/log/apache2/error.log"
flags(no-parse));
file("/var/log/apache2/other_vhosts_access.log"
flags(no-parse));
file("/var/log/apache2/ssl_access.log"
flags(no-parse));
file("/var/log/apache2/suexec.log"
flags(no-parse));
};
destination d_shell_access_system {
syslog(
"shell-system.domain.com"
transport("tls")
port(6514)
tls(
peer-verify(required-trusted)
ca_dir('/etc/syslog-ng/ssl/ca.d')
key_file('/etc/syslog-ng/ssl/server.key')
cert_file('/etc/syslog-ng/ssl/server.crt')
)
);
};
# The funny .SDATA.file at 18372.4.name is for structured data which, I
believe, is part of the (new)
# syslog protocol - RFC5424-formatted (IETF-syslog).
# We need to set the filename value in the SDATA field. $FILE_NAME is
a macro which returns the
# full filename path. ie '/var/log/apache2/access.log'. That will then
get assigned to the
# structured data value, .SDATA.file at 18372.4.name .
rewrite r_setfilename {
set(
"$FILE_NAME",
value(".SDATA.file at 18372.4.name")
);
};
# After getting the value set for the filename, truncate the directory
portion and only use the
# basename. Use a simple string substitution.
rewrite r_use_basename {
subst(
"/var/log/apache2/",
"",
value(".SDATA.file at 18372.4.name")
type("string")
flags("prefix")
);
};
log {
source(s_apache_logs);
rewrite(r_setfilename);
rewrite(r_use_basename);
destination(d_shell_access_system);
};
---{END OF web server syslog-ng configs}---
---{shell access system syslog-ng configs}---
source s_tls {
syslog(
ip(0.0.0.0)
port(6514)
transport("tls")
tls(
peer-verify(required-trusted)
ca_dir('/etc/syslog-ng/ssl/ca.d')
key_file('/etc/syslog-ng/ssl/server.key')
cert_file('/etc/syslog-ng/ssl/server.crt')
)
max_connections(1000)
keep_hostname(yes)
);
};
destination d_web_log_files {
file(
"/usr/local/apache2/log/${.SDATA.file at 18372.4.name}"
create_dirs(yes)
);
};
filter f_web_server {
host("webserver");
};
log {
source(s_tls);
filter(f_web_server);
destination(d_weblog_files);
};
---{END OF shell access system syslog-ng configs}---
More information about the syslog-ng
mailing list