[syslog-ng] preserving tags across the network

[Fekete Róbert]

On Friday, November 11, 2011 18:04 CET, Matt Zagrabelny <mzagrabe at d.umn.edu> wrote: 
 
> Hi,
> 
> I am trying to ship a bunch of apache log files across the network and
> on the syslog-ng server side then break them into their individual
> files again.
> 
> I am not sure the best way to do this, but it looks like tags might be helpful.

Hi, tags are not part of the syslog message unless you add them to the message using a template on the client, and then somehow extract them from the messages on the server. But it seems that you are actually trying to separate logs from different files, and you are using the syslog() drivers on your server and clients. If you use the file source and the syslog destination, syslog-ng adds the filename and some other metadata to the SDATA part of the message. You can extract this on the server side, and use it as a macro in the filename template on your server.
Like: 
destination d_test { 
   file( "/var/log/apache2/${.SDATA.file at 18372.4.name}" 
          create_dirs(yes)
        ); 
};

For details on other metadata added to SDATA, see http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides/syslog-ng-pe-v4.0-guide-admin-en.html/file-source-and-syslog-destination.html

Note to myself: this section is missing from the OSE guide for some reason, even though I believe the feature is available in OSE. Should check with Bazsi and update the docs.

Regards, 

Robert

> 
> On my apache system I have the following snippets:
> 
> -----{syslog-ng client begin}-----
> 
> source s_apache_access {
>   file("/var/log/apache2/access.log"
>        tags("main_access")
>       );
> };
> 
> source s_apache_other_vhost_access {
>   file("/var/log/apache2/other_vhost.log"
>        tags("other_vhost_access")
>       );
> };
> 
> destination d_server {
>                    syslog(
>                           "10.0.0.1"
>                           transport("tls")
>                           port(6514)
>                           tls(
>                               peer-verify(required-trusted)
>                               ca_dir('/etc/syslog-ng/ssl/ca.d')
>                               key_file('/etc/syslog-ng/ssl/server.key')
>                               cert_file('/etc/syslog-ng/ssl/server.crt')
>                              )
>                          );
> };
> 
> log {
>   source(s_apache_access);
>   source(s_apache_other_vhost_access);
>   destination(d_server);
> };
> 
> -----{syslog-ng client end}-----
> 
> On my server I have the following:
> 
> -----{syslog-ng server begin}-----
> 
> source s_tls {
>               syslog(
>                      ip(0.0.0.0)
>                      port(6514)
>                      transport("tls")
>                      tls(
>                          peer-verify(required-trusted)
>                          ca_dir('/etc/syslog-ng/ssl/ca.d')
>                          key_file('/etc/syslog-ng/ssl/server.key')
>                          cert_file('/etc/syslog-ng/ssl/server.crt')
>                         )
>                      max_connections(1000)
>                      keep_hostname(yes)
>                     );
> };
> 
> filter f_main_apache_access {
>   tags("main_access");
> };
> 
> destination d_main_access {
>   file("/var/log/apache2/access.log");
> };
> 
> log {
>   source(s_tls);
>   filter(f_main_apache_access);
>   destination(d_main_access);
> };
> 
> -----{syslog-ng server end}-----
> 
> It doesn't look like the "tag" is being preserved through the network.
> Does anyone know if tags is the right mechanism for splitting apart
> (filtering) the aggregate stream coming in on the server?
> 
> Any other advice would be greatly appreciated.
> 
> Thanks!
> 
> -matt zagrabelny
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
>