preserving tags across the network
Hi, I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again. I am not sure the best way to do this, but it looks like tags might be helpful. On my apache system I have the following snippets: -----{syslog-ng client begin}----- source s_apache_access { file("/var/log/apache2/access.log" tags("main_access") ); }; source s_apache_other_vhost_access { file("/var/log/apache2/other_vhost.log" tags("other_vhost_access") ); }; destination d_server { syslog( "10.0.0.1" transport("tls") port(6514) tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) ); }; log { source(s_apache_access); source(s_apache_other_vhost_access); destination(d_server); }; -----{syslog-ng client end}----- On my server I have the following: -----{syslog-ng server begin}----- source s_tls { syslog( ip(0.0.0.0) port(6514) transport("tls") tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) max_connections(1000) keep_hostname(yes) ); }; filter f_main_apache_access { tags("main_access"); }; destination d_main_access { file("/var/log/apache2/access.log"); }; log { source(s_tls); filter(f_main_apache_access); destination(d_main_access); }; -----{syslog-ng server end}----- It doesn't look like the "tag" is being preserved through the network. Does anyone know if tags is the right mechanism for splitting apart (filtering) the aggregate stream coming in on the server? Any other advice would be greatly appreciated. Thanks! -matt zagrabelny
On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Hi,
I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again.
I am not sure the best way to do this, but it looks like tags might be helpful.
Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server. Like: destination d_test { file( "/var/log/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); }; For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides... Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs. Regards, Robert
On my apache system I have the following snippets:
-----{syslog-ng client begin}-----
source s_apache_access { file("/var/log/apache2/access.log" tags("main_access") ); };
source s_apache_other_vhost_access { file("/var/log/apache2/other_vhost.log" tags("other_vhost_access") ); };
destination d_server { syslog( "10.0.0.1" transport("tls") port(6514) tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) ); };
log { source(s_apache_access); source(s_apache_other_vhost_access); destination(d_server); };
-----{syslog-ng client end}-----
On my server I have the following:
-----{syslog-ng server begin}-----
source s_tls { syslog( ip(0.0.0.0) port(6514) transport("tls") tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) max_connections(1000) keep_hostname(yes) ); };
filter f_main_apache_access { tags("main_access"); };
destination d_main_access { file("/var/log/apache2/access.log"); };
log { source(s_tls); filter(f_main_apache_access); destination(d_main_access); };
-----{syslog-ng server end}-----
It doesn't look like the "tag" is being preserved through the network. Does anyone know if tags is the right mechanism for splitting apart (filtering) the aggregate stream coming in on the server?
Any other advice would be greatly appreciated.
Thanks!
-matt zagrabelny ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, Nov 11, 2011 at 1:23 PM, Fekete Róbert <frobert@balabit.hu> wrote:
On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Hi,
I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again.
I am not sure the best way to do this, but it looks like tags might be helpful.
Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server. Like: destination d_test { file( "/var/log/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides...
Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs.
Hi Róbert, Thanks for the hints. I am using syslog-ng from Debian Squeeze: ii syslog-ng 3.1.3-3 Next generation logging daemon I have the following configured now on the server: destination d_web2_access { file( "/tmp/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); }; log { source(s_tls); destination(d_web2_access); }; But all I see is the (newly created) /tmp/apache2 directory. I can verify that I am getting apache logs sent over because they are making it into the "user" facility file for my generic logging. The relevant config snippet is: destination d_remote_clients { file( "/var/log/syslog-ng/remote_clients/$HOST_FROM/$YEAR/$MONTH/$DAY/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; Any other ideas why the SDATA macro is not working? Thanks again for the help! -matt zagrabelny
On Fri, 2011-11-11 at 14:01 -0600, Matt Zagrabelny wrote:
On Fri, Nov 11, 2011 at 1:23 PM, Fekete Róbert <frobert@balabit.hu> wrote:
On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Hi,
I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again.
I am not sure the best way to do this, but it looks like tags might be helpful.
Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server. Like: destination d_test { file( "/var/log/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides...
Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs.
Hi Róbert,
Thanks for the hints.
I am using syslog-ng from Debian Squeeze:
ii syslog-ng 3.1.3-3 Next generation logging daemon
I have the following configured now on the server:
destination d_web2_access { file( "/tmp/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
log { source(s_tls); destination(d_web2_access); };
But all I see is the (newly created) /tmp/apache2 directory.
I can verify that I am getting apache logs sent over because they are making it into the "user" facility file for my generic logging. The relevant config snippet is:
destination d_remote_clients { file( "/var/log/syslog-ng/remote_clients/$HOST_FROM/$YEAR/$MONTH/$DAY/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); };
Any other ideas why the SDATA macro is not working?
Thanks again for the help!
I think the automatic addition of the filename into the SDATA is part of the wild-card file source support of the PE version. The code itself is available in the syslog-ng team's public git repository (git.balabit.hu), but I didn't have the time to do a proper review & integration on that code. So this feature is PE only right now, but the code is public, the roadblock is to get enough time on my part, which seems never to happen :( The wild-card file source is a complex piece of code, and I don't want to blindly pull it, since then all the support burden is on me. Anyway, an alternative is to explicitly set the .SDATA field on your client side, as the name of the file is available in the $FILE_NAME macro. This rewrite rule puts the same value in the same SDATA field as the PE does (and the OSE will, once the code gets integrated): rewrite r_setfilename { set("$FILE_NAME" value(".SDATA.file@18374.4.name")); This should work in 3.1 clients too. -- Bazsi
2011/11/12 Balazs Scheidler <bazsi@balabit.hu>:
On Fri, 2011-11-11 at 14:01 -0600, Matt Zagrabelny wrote:
On Fri, Nov 11, 2011 at 1:23 PM, Fekete Róbert <frobert@balabit.hu> wrote:
On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Hi,
I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again.
I am not sure the best way to do this, but it looks like tags might be helpful.
Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server. Like: destination d_test { file( "/var/log/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides...
Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs.
Hi Róbert,
Thanks for the hints.
I am using syslog-ng from Debian Squeeze:
ii syslog-ng 3.1.3-3 Next generation logging daemon
I have the following configured now on the server:
destination d_web2_access { file( "/tmp/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
log { source(s_tls); destination(d_web2_access); };
But all I see is the (newly created) /tmp/apache2 directory.
I can verify that I am getting apache logs sent over because they are making it into the "user" facility file for my generic logging. The relevant config snippet is:
destination d_remote_clients { file( "/var/log/syslog-ng/remote_clients/$HOST_FROM/$YEAR/$MONTH/$DAY/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); };
Any other ideas why the SDATA macro is not working?
Thanks again for the help!
I think the automatic addition of the filename into the SDATA is part of the wild-card file source support of the PE version. The code itself is available in the syslog-ng team's public git repository (git.balabit.hu), but I didn't have the time to do a proper review & integration on that code.
So this feature is PE only right now, but the code is public, the roadblock is to get enough time on my part, which seems never to happen :( The wild-card file source is a complex piece of code, and I don't want to blindly pull it, since then all the support burden is on me.
Anyway, an alternative is to explicitly set the .SDATA field on your client side, as the name of the file is available in the $FILE_NAME macro.
This rewrite rule puts the same value in the same SDATA field as the PE does (and the OSE will, once the code gets integrated):
rewrite r_setfilename { set("$FILE_NAME" value(".SDATA.file@18374.4.name"));
This should work in 3.1 clients too.
Super. Thanks for the hints, Bazsi. (PS. I think there was a typo in the 'file@18374.4' above. I think it should be 'file@18372.4'.) -matt zagrabelny For anyone grepping the mailing list archive, here are my relevant snippets: ---{web server syslog-ng configs}--- # Send the following apache log files to the shell access system so that people with shell access can examine # log messages. source s_apache_logs { file("/var/log/apache2/access.log" flags(no-parse)); file("/var/log/apache2/error.log" flags(no-parse)); file("/var/log/apache2/other_vhosts_access.log" flags(no-parse)); file("/var/log/apache2/ssl_access.log" flags(no-parse)); file("/var/log/apache2/suexec.log" flags(no-parse)); }; destination d_shell_access_system { syslog( "shell-system.domain.com" transport("tls") port(6514) tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) ); }; # The funny .SDATA.file@18372.4.name is for structured data which, I believe, is part of the (new) # syslog protocol - RFC5424-formatted (IETF-syslog). # We need to set the filename value in the SDATA field. $FILE_NAME is a macro which returns the # full filename path. ie '/var/log/apache2/access.log'. That will then get assigned to the # structured data value, .SDATA.file@18372.4.name . rewrite r_setfilename { set( "$FILE_NAME", value(".SDATA.file@18372.4.name") ); }; # After getting the value set for the filename, truncate the directory portion and only use the # basename. Use a simple string substitution. rewrite r_use_basename { subst( "/var/log/apache2/", "", value(".SDATA.file@18372.4.name") type("string") flags("prefix") ); }; log { source(s_apache_logs); rewrite(r_setfilename); rewrite(r_use_basename); destination(d_shell_access_system); }; ---{END OF web server syslog-ng configs}--- ---{shell access system syslog-ng configs}--- source s_tls { syslog( ip(0.0.0.0) port(6514) transport("tls") tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) max_connections(1000) keep_hostname(yes) ); }; destination d_web_log_files { file( "/usr/local/apache2/log/${.SDATA.file@18372.4.name}" create_dirs(yes) ); }; filter f_web_server { host("webserver"); }; log { source(s_tls); filter(f_web_server); destination(d_weblog_files); }; ---{END OF shell access system syslog-ng configs}---
On Thu, 2011-11-17 at 08:36 -0600, Matt Zagrabelny wrote:
2011/11/12 Balazs Scheidler <bazsi@balabit.hu>:
On Fri, 2011-11-11 at 14:01 -0600, Matt Zagrabelny wrote:
On Fri, Nov 11, 2011 at 1:23 PM, Fekete Róbert <frobert@balabit.hu> wrote:
On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Hi,
I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again.
I am not sure the best way to do this, but it looks like tags might be helpful.
Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server. Like: destination d_test { file( "/var/log/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides...
Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs.
Hi Róbert,
Thanks for the hints.
I am using syslog-ng from Debian Squeeze:
ii syslog-ng 3.1.3-3 Next generation logging daemon
I have the following configured now on the server:
destination d_web2_access { file( "/tmp/apache2/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
log { source(s_tls); destination(d_web2_access); };
But all I see is the (newly created) /tmp/apache2 directory.
I can verify that I am getting apache logs sent over because they are making it into the "user" facility file for my generic logging. The relevant config snippet is:
destination d_remote_clients { file( "/var/log/syslog-ng/remote_clients/$HOST_FROM/$YEAR/$MONTH/$DAY/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); };
Any other ideas why the SDATA macro is not working?
Thanks again for the help!
I think the automatic addition of the filename into the SDATA is part of the wild-card file source support of the PE version. The code itself is available in the syslog-ng team's public git repository (git.balabit.hu), but I didn't have the time to do a proper review & integration on that code.
So this feature is PE only right now, but the code is public, the roadblock is to get enough time on my part, which seems never to happen :( The wild-card file source is a complex piece of code, and I don't want to blindly pull it, since then all the support burden is on me.
Anyway, an alternative is to explicitly set the .SDATA field on your client side, as the name of the file is available in the $FILE_NAME macro.
This rewrite rule puts the same value in the same SDATA field as the PE does (and the OSE will, once the code gets integrated):
rewrite r_setfilename { set("$FILE_NAME" value(".SDATA.file@18374.4.name"));
This should work in 3.1 clients too.
Super. Thanks for the hints, Bazsi. (PS. I think there was a typo in the 'file@18374.4' above. I think it should be 'file@18372.4'.)
-matt zagrabelny
For anyone grepping the mailing list archive, here are my relevant snippets:
---{web server syslog-ng configs}--- # Send the following apache log files to the shell access system so that people with shell access can examine # log messages. source s_apache_logs { file("/var/log/apache2/access.log" flags(no-parse)); file("/var/log/apache2/error.log" flags(no-parse)); file("/var/log/apache2/other_vhosts_access.log" flags(no-parse)); file("/var/log/apache2/ssl_access.log" flags(no-parse)); file("/var/log/apache2/suexec.log" flags(no-parse)); };
destination d_shell_access_system { syslog( "shell-system.domain.com" transport("tls") port(6514) tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) ); };
# The funny .SDATA.file@18372.4.name is for structured data which, I believe, is part of the (new) # syslog protocol - RFC5424-formatted (IETF-syslog).
# We need to set the filename value in the SDATA field. $FILE_NAME is a macro which returns the # full filename path. ie '/var/log/apache2/access.log'. That will then get assigned to the # structured data value, .SDATA.file@18372.4.name .
rewrite r_setfilename { set( "$FILE_NAME", value(".SDATA.file@18372.4.name") ); };
# After getting the value set for the filename, truncate the directory portion and only use the # basename. Use a simple string substitution.
rewrite r_use_basename { subst( "/var/log/apache2/", "", value(".SDATA.file@18372.4.name") type("string") flags("prefix") ); };
note that you can use several rewrite expression in the same rewrite block, like: rewrite r_setfilename { set( "$FILE_NAME", value(".SDATA.file@18372.4.name") ); subst( "/var/log/apache2/", "", value(".SDATA.file@18372.4.name") type("string") flags("prefix")); }; This will apply both rewrite expressions one after the other.
log { source(s_apache_logs); rewrite(r_setfilename); rewrite(r_use_basename); destination(d_shell_access_system); }; ---{END OF web server syslog-ng configs}---
---{shell access system syslog-ng configs}--- source s_tls { syslog( ip(0.0.0.0) port(6514) transport("tls") tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) max_connections(1000) keep_hostname(yes) ); };
destination d_web_log_files { file( "/usr/local/apache2/log/${.SDATA.file@18372.4.name}" create_dirs(yes) ); };
Please note what happens if the SDATA portion contains '../' or '/..'. syslog-ng will refuse to open files if that happens. Also, in the 3.4 tree there's a $(sanitize) template function: * $(sanitize [opts] $arg1 $arg2 ...) * * Options: * --ctrl-chars or -c Filter control characters (default) * --no-ctrl-chars or -C Don't filter control characters * --invalid-chars <set> or -i Set of characters to be translated, default "/" * --replacement <replace> or -r Single character replacement for invalid chars. -- Bazsi
Le 11 nov. 2011 17:06, "Matt Zagrabelny" <mzagrabe@d.umn.edu> a écrit :
Hi,
I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again.
I am not sure the best way to do this, but it looks like tags might be helpful.
On my apache system I have the following snippets:
-----{syslog-ng client begin}-----
source s_apache_access { file("/var/log/apache2/access.log" tags("main_access") ); };
source s_apache_other_vhost_access { file("/var/log/apache2/other_vhost.log" tags("other_vhost_access") ); };
destination d_server { syslog( "10.0.0.1" transport("tls") port(6514) tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) ); };
log { source(s_apache_access); source(s_apache_other_vhost_access); destination(d_server); };
-----{syslog-ng client end}-----
On my server I have the following:
-----{syslog-ng server begin}-----
source s_tls { syslog( ip(0.0.0.0) port(6514) transport("tls") tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) max_connections(1000) keep_hostname(yes) ); };
filter f_main_apache_access { tags("main_access"); };
destination d_main_access { file("/var/log/apache2/access.log"); };
log { source(s_tls); filter(f_main_apache_access); destination(d_main_access); };
-----{syslog-ng server end}-----
It doesn't look like the "tag" is being preserved through the network. Does anyone know if tags is the right mechanism for splitting apart (filtering) the aggregate stream coming in on the server?
Any other advice would be greatly appreciated.
Thanks!
-matt zagrabelny
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balazs Scheidler
-
Fekete Róbert
-
Joël Landry Nyamsi Kouendé
-
Matt Zagrabelny