Problem using tags with syslog-ng 3.1
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this: ------------------------------------------------------------------------ ---------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { }; destination d_app { tcp("logrelay01.domain.com" port(13074)); }; log { source(s_app); destination(d_app); }; ------------------------------------------------------------------------ ---------------------------- The log relay does nothing than forward the messages to the central logging server with the following config: ------------------------------------------------------------------------ ---------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); }; source s_remote { tcp(ip("0.0.0.0") port(13074)); }; destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); }; log { source(s_remote); destination(remote_tcp); }; ------------------------------------------------------------------------ ---------------------------- On the central logging server I use filters to separate the logfiles again: ------------------------------------------------------------------------ ---------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf"; options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); }; source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================ This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); }; #Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); }; # Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); }; # Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); }; ------------------------------------------------------------------------ ---------------------------- Does anybody have an idea, why it does not work as expected. Thanks for any suggestion and/or idea. Best regards Stefan
Hi, I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng. However, I can suggest you an other solution: use the program_override option. This will override the $PROGRAM macro with the specified value. For example: source s_app { file("/var/log/log1.log" program_override("/var/log/log1.log")); file("/opt//log/log2.log" tags("log2") program_override("/opt/log/log2.log")); file("/opt/log/log3.log" tags("log3") program_override("/opt/log/log3.log")); }; After that, you can use a specified program filter on the central logging server side to separate them. 2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this: ---------------------------------------------------------------------------------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { };
destination d_app { tcp("logrelay01.domain.com" port(13074)); };
log { source(s_app); destination(d_app); }; ---------------------------------------------------------------------------------------------------- The log relay does nothing than forward the messages to the central logging server with the following config: ---------------------------------------------------------------------------------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); };
source s_remote { tcp(ip("0.0.0.0") port(13074)); }; destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); }; log { source(s_remote); destination(remote_tcp); }; ---------------------------------------------------------------------------------------------------- On the central logging server I use filters to separate the logfiles again: ---------------------------------------------------------------------------------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf"; options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); };
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================ This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); }; #Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); }; # Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); }; # Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); }; ---------------------------------------------------------------------------------------------------- Does anybody have an idea, why it does not work as expected. Thanks for any suggestion and/or idea. Best regards Stefan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax. Namely, using tags(".source.log2") in your filter. On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Hi,
I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.
However, I can suggest you an other solution: use the program_override option. This will override the $PROGRAM macro with the specified value. For example: source s_app { file("/var/log/log1.log" program_override("/var/log/log1.log")); file("/opt//log/log2.log" tags("log2") program_override(" /opt/log/log2.log")); file("/opt/log/log3.log" tags("log3") program_override("/opt/log/log3.log" )); };
After that, you can use a specified program filter on the central logging server side to separate them.
2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this:
---------------------------------------------------------------------------------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { };
destination d_app { tcp("logrelay01.domain.com" port(13074)); };
log { source(s_app); destination(d_app); };
----------------------------------------------------------------------------------------------------
The log relay does nothing than forward the messages to the central logging server with the following config:
---------------------------------------------------------------------------------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); };
source s_remote { tcp(ip("0.0.0.0") port(13074)); };
destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); };
log { source(s_remote); destination(remote_tcp); };
----------------------------------------------------------------------------------------------------
On the central logging server I use filters to separate the logfiles again:
---------------------------------------------------------------------------------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf";
options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); };
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================
This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); };
#Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); };
# Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); };
# Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); };
----------------------------------------------------------------------------------------------------
Does anybody have an idea, why it does not work as expected.
Thanks for any suggestion and/or idea.
Best regards Stefan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Martin Holste wrote:
Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax. Namely, using tags(".source.log2") in your filter. No, you are using a wrong tag name. In this case, you can use the following tag: tags(".source.s_app")
this is an on-the-fly generated tag,every incoming message has it one given with the following formula: ".source.<sourcename>"
On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee@balabit.hu <mailto:pzolee@balabit.hu>> wrote:
Hi,
I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.
However, I can suggest you an other solution: use the program_override option. This will override the $PROGRAM macro with the specified value. For example: source s_app { file("/var/log/log1.log" program_override("/var/log/log1.log")); file("/opt//log/log2.log" tags("log2") program_override("/opt/log/log2.log")); file("/opt/log/log3.log" tags("log3") program_override("/opt/log/log3.log")); };
After that, you can use a specified program filter on the central logging server side to separate them.
2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this: ---------------------------------------------------------------------------------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { };
destination d_app { tcp("logrelay01.domain.com <http://logrelay01.domain.com>" port(13074)); };
log { source(s_app); destination(d_app); }; ----------------------------------------------------------------------------------------------------
The log relay does nothing than forward the messages to the central logging server with the following config: ---------------------------------------------------------------------------------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); };
source s_remote { tcp(ip("0.0.0.0") port(13074)); };
destination remote_tcp { tcp("centrallog01.domain.com <http://centrallog01.domain.com>" port(13074)); };
log { source(s_remote); destination(remote_tcp); }; ----------------------------------------------------------------------------------------------------
On the central logging server I use filters to separate the logfiles again: ---------------------------------------------------------------------------------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf";
options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); };
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================
This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); };
#Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); };
# Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); };
# Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); }; ----------------------------------------------------------------------------------------------------
Does anybody have an idea, why it does not work as expected.
Thanks for any suggestion and/or idea.
Best regards Stefan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Ok, so '.sources' has nothing to do with user-provided tags, but in his example Marci uses: source s_tcp2 { tcp(ip(192.168.1.2) port(1514) tags("tcp", "windows)); }; #Match on tags "tcp" or "udp" filter f_net { tags("tcp", "udp"); }; Which seems to imply that arbitrary user tags can be set and then matched on in a filter later. So, it seems what is missing from Stefan's config was: source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes) tags("log2"); }; Which would allow his later filter statement filter f_log2 { host("web00(09|10)") and tags("log2"); }; to succeed. Right? On Thu, Apr 1, 2010 at 8:37 AM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Martin Holste wrote:
Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax. Namely, using tags(".source.log2") in your filter.
No, you are using a wrong tag name. In this case, you can use the following tag: tags(".source.s_app")
this is an on-the-fly generated tag,every incoming message has it one given with the following formula: ".source.<sourcename>"
On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee@balabit.hu>wrote:
Hi,
I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.
However, I can suggest you an other solution: use the program_override option. This will override the $PROGRAM macro with the specified value. For example: source s_app { file("/var/log/log1.log" program_override("/var/log/log1.log")); file("/opt//log/log2.log" tags("log2") program_override(" /opt/log/log2.log")); file("/opt/log/log3.log" tags("log3") program_override(" /opt/log/log3.log")); };
After that, you can use a specified program filter on the central logging server side to separate them.
2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this:
---------------------------------------------------------------------------------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { };
destination d_app { tcp("logrelay01.domain.com" port(13074)); };
log { source(s_app); destination(d_app); };
----------------------------------------------------------------------------------------------------
The log relay does nothing than forward the messages to the central logging server with the following config:
---------------------------------------------------------------------------------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); };
source s_remote { tcp(ip("0.0.0.0") port(13074)); };
destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); };
log { source(s_remote); destination(remote_tcp); };
----------------------------------------------------------------------------------------------------
On the central logging server I use filters to separate the logfiles again:
---------------------------------------------------------------------------------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf";
options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); };
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================
This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); };
#Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); };
# Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); };
# Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); };
----------------------------------------------------------------------------------------------------
Does anybody have an idea, why it does not work as expected.
Thanks for any suggestion and/or idea.
Best regards Stefan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Martin Holste wrote:
Ok, so '.sources' has nothing to do with user-provided tags, but in his example Marci uses:
source s_tcp2 { tcp(ip(192.168.1.2) port(1514) tags("tcp", "windows)); };
#Match on tags "tcp" or "udp" filter f_net { tags("tcp", "udp"); };
Which seems to imply that arbitrary user tags can be set and then matched on in a filter later. So, it seems what is missing from Stefan's config was:
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes) tags("log2"); };
Which would allow his later filter statement
filter f_log2 { host("web00(09|10)") and tags("log2"); };
to succeed. Right? Yes and no. It's a correct way to use tagging on sources.
And no, because his original aim wasn't that. He wanted to mark logs coming from different source files of the client by tags and than to recreate almost the same file and log structure on the server by these tags. In your solution he would have only one tag for all the messages coming from different sources of client.
On Thu, Apr 1, 2010 at 8:37 AM, Zoltán Pallagi <pzolee@balabit.hu <mailto:pzolee@balabit.hu>> wrote:
Martin Holste wrote:
Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax. Namely, using tags(".source.log2") in your filter.
No, you are using a wrong tag name. In this case, you can use the following tag: tags(".source.s_app")
this is an on-the-fly generated tag,every incoming message has it one given with the following formula: ".source.<sourcename>"
On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee@balabit.hu <mailto:pzolee@balabit.hu>> wrote:
Hi,
I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.
However, I can suggest you an other solution: use the program_override option. This will override the $PROGRAM macro with the specified value. For example: source s_app { file("/var/log/log1.log" program_override("/var/log/log1.log")); file("/opt//log/log2.log" tags("log2") program_override("/opt/log/log2.log")); file("/opt/log/log3.log" tags("log3") program_override("/opt/log/log3.log")); };
After that, you can use a specified program filter on the central logging server side to separate them.
2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this: ---------------------------------------------------------------------------------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { };
destination d_app { tcp("logrelay01.domain.com <http://logrelay01.domain.com>" port(13074)); };
log { source(s_app); destination(d_app); }; ----------------------------------------------------------------------------------------------------
The log relay does nothing than forward the messages to the central logging server with the following config: ---------------------------------------------------------------------------------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); };
source s_remote { tcp(ip("0.0.0.0") port(13074)); };
destination remote_tcp { tcp("centrallog01.domain.com <http://centrallog01.domain.com>" port(13074)); };
log { source(s_remote); destination(remote_tcp); }; ----------------------------------------------------------------------------------------------------
On the central logging server I use filters to separate the logfiles again: ---------------------------------------------------------------------------------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf";
options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); };
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================
This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); };
#Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); };
# Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); };
# Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); }; ----------------------------------------------------------------------------------------------------
Does anybody have an idea, why it does not work as expected.
Thanks for any suggestion and/or idea.
Best regards Stefan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Ok, I think I see what you're saying: the tag only exists on the local box and does not get forwarded in the message. You were saying you have to overwrite the original program with some other value so that the tag is permanent and will survive multiple relays. Sorry for the confusion. So the difference is that tags are only supported on some types of sources? On Thu, Apr 1, 2010 at 9:40 AM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Martin Holste wrote:
Ok, so '.sources' has nothing to do with user-provided tags, but in his example Marci uses:
source s_tcp2 { tcp(ip(192.168.1.2) port(1514) tags("tcp", "windows)); };
#Match on tags "tcp" or "udp" filter f_net { tags("tcp", "udp"); };
Which seems to imply that arbitrary user tags can be set and then matched on in a filter later. So, it seems what is missing from Stefan's config was:
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes) tags("log2"); };
Which would allow his later filter statement
filter f_log2 { host("web00(09|10)") and tags("log2"); };
to succeed. Right?
Yes and no. It's a correct way to use tagging on sources.
And no, because his original aim wasn't that. He wanted to mark logs coming from different source files of the client by tags and than to recreate almost the same file and log structure on the server by these tags.
In your solution he would have only one tag for all the messages coming from different sources of client.
On Thu, Apr 1, 2010 at 8:37 AM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Martin Holste wrote:
Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax. Namely, using tags(".source.log2") in your filter.
No, you are using a wrong tag name. In this case, you can use the following tag: tags(".source.s_app")
this is an on-the-fly generated tag,every incoming message has it one given with the following formula: ".source.<sourcename>"
On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee@balabit.hu>wrote:
Hi,
I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.
However, I can suggest you an other solution: use the program_override option. This will override the $PROGRAM macro with the specified value. For example: source s_app { file("/var/log/log1.log" program_override("/var/log/log1.log")); file("/opt//log/log2.log" tags("log2") program_override(" /opt/log/log2.log")); file("/opt/log/log3.log" tags("log3") program_override(" /opt/log/log3.log")); };
After that, you can use a specified program filter on the central logging server side to separate them.
2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this:
---------------------------------------------------------------------------------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { };
destination d_app { tcp("logrelay01.domain.com" port(13074)); };
log { source(s_app); destination(d_app); };
----------------------------------------------------------------------------------------------------
The log relay does nothing than forward the messages to the central logging server with the following config:
---------------------------------------------------------------------------------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); };
source s_remote { tcp(ip("0.0.0.0") port(13074)); };
destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); };
log { source(s_remote); destination(remote_tcp); };
----------------------------------------------------------------------------------------------------
On the central logging server I use filters to separate the logfiles again:
---------------------------------------------------------------------------------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf";
options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); };
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================
This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); };
#Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); };
# Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); };
# Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); };
----------------------------------------------------------------------------------------------------
Does anybody have an idea, why it does not work as expected.
Thanks for any suggestion and/or idea.
Best regards Stefan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Martin Holste wrote:
Ok, I think I see what you're saying: the tag only exists on the local box and does not get forwarded in the message. You were saying you have to overwrite the original program with some other value so that the tag is permanent and will survive multiple relays. Sorry for the confusion.
So the difference is that tags are only supported on some types of sources? Please, read our guide about tagging and you will find the answers of your questions: http://www.balabit.hu/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/ch04s06...
The only missing part is that the sent(or forwarded) message will not contain these tags (so it's not a macro and you cannot use them in a template string). I will ask our tech writer to include it into the documentation.
On Thu, Apr 1, 2010 at 9:40 AM, Zoltán Pallagi <pzolee@balabit.hu <mailto:pzolee@balabit.hu>> wrote:
Martin Holste wrote:
Ok, so '.sources' has nothing to do with user-provided tags, but in his example Marci uses:
source s_tcp2 { tcp(ip(192.168.1.2) port(1514) tags("tcp", "windows)); };
#Match on tags "tcp" or "udp" filter f_net { tags("tcp", "udp"); };
Which seems to imply that arbitrary user tags can be set and then matched on in a filter later. So, it seems what is missing from Stefan's config was:
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes) tags("log2"); };
Which would allow his later filter statement
filter f_log2 { host("web00(09|10)") and tags("log2"); };
to succeed. Right?
Yes and no. It's a correct way to use tagging on sources.
And no, because his original aim wasn't that. He wanted to mark logs coming from different source files of the client by tags and than to recreate almost the same file and log structure on the server by these tags.
In your solution he would have only one tag for all the messages coming from different sources of client.
On Thu, Apr 1, 2010 at 8:37 AM, Zoltán Pallagi <pzolee@balabit.hu <mailto:pzolee@balabit.hu>> wrote:
Martin Holste wrote:
Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax. Namely, using tags(".source.log2") in your filter.
No, you are using a wrong tag name. In this case, you can use the following tag: tags(".source.s_app")
this is an on-the-fly generated tag,every incoming message has it one given with the following formula: ".source.<sourcename>"
On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee@balabit.hu <mailto:pzolee@balabit.hu>> wrote:
Hi,
I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.
However, I can suggest you an other solution: use the program_override option. This will override the $PROGRAM macro with the specified value. For example: source s_app { file("/var/log/log1.log" program_override("/var/log/log1.log")); file("/opt//log/log2.log" tags("log2") program_override("/opt/log/log2.log")); file("/opt/log/log3.log" tags("log3") program_override("/opt/log/log3.log")); };
After that, you can use a specified program filter on the central logging server side to separate them.
2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:
Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this: ---------------------------------------------------------------------------------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { };
destination d_app { tcp("logrelay01.domain.com <http://logrelay01.domain.com>" port(13074)); };
log { source(s_app); destination(d_app); }; ----------------------------------------------------------------------------------------------------
The log relay does nothing than forward the messages to the central logging server with the following config: ---------------------------------------------------------------------------------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); };
source s_remote { tcp(ip("0.0.0.0") port(13074)); };
destination remote_tcp { tcp("centrallog01.domain.com <http://centrallog01.domain.com>" port(13074)); };
log { source(s_remote); destination(remote_tcp); }; ----------------------------------------------------------------------------------------------------
On the central logging server I use filters to separate the logfiles again: ---------------------------------------------------------------------------------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf";
options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); };
source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================
This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); };
#Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); };
# Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); };
# Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); }; ----------------------------------------------------------------------------------------------------
Does anybody have an idea, why it does not work as expected.
Thanks for any suggestion and/or idea.
Best regards Stefan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
------------------------------------------------------------------------ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Hi All! On Thu, 2010-04-01 at 11:37 -0600, Martin Holste wrote:
Ok, I think I see what you're saying: the tag only exists on the local box and does not get forwarded in the message. You were saying you have to overwrite the original program with some other value so that the tag is permanent and will survive multiple relays. Sorry for the confusion.
Try to think about the tags as a sticky note on a package. When I want to create a lot of uniform white bag I put a sticky note into it, so I can make a difference between them. But after I write the correct adresses I take off the notes from it. In the example the tag is not surviving the relays, He put the tag _value_ into an another field. Just like if I put some information into the destination address to my mail. But I think that the idea of the persistant tags is great. And in the new syslog protocol there is space for it. I will create a feature request for it. :)
On Fri, Apr 2, 2010 at 1:25 AM, SZALAY Attila <sasa@balabit.hu> wrote:
Hi All!
On Thu, 2010-04-01 at 11:37 -0600, Martin Holste wrote:
Ok, I think I see what you're saying: the tag only exists on the local box and does not get forwarded in the message. You were saying you have to overwrite the original program with some other value so that the tag is permanent and will survive multiple relays. Sorry for the confusion.
Try to think about the tags as a sticky note on a package. When I want to create a lot of uniform white bag I put a sticky note into it, so I can make a difference between them. But after I write the correct adresses I take off the notes from it. In the example the tag is not surviving the relays, He put the tag _value_ into an another field. Just like if I put some information into the destination address to my mail.
But I think that the idea of the persistant tags is great. And in the new syslog protocol there is space for it. I will create a feature request for it. :)
I agree. When tagging was first announced I was disappointed that they would not survive relays. Program_override is a way to 'tag' relayed messages, but it seems like a work-around (just like, say, using templates to add a custom string before $MSG to filter for at your receiving host). In a large setup, matching by host or other means is not manageable (especially if you want your configs to be multi-site compatible), so I think persistent tags would be a useful feature when sending from syslog-ng to syslog-ng. -- Lance Laursen Demonware Systems Engineer
On Fri, 2010-04-02 at 15:50 -0700, Lance Laursen wrote:
On Fri, Apr 2, 2010 at 1:25 AM, SZALAY Attila <sasa@balabit.hu> wrote: Hi All!
On Thu, 2010-04-01 at 11:37 -0600, Martin Holste wrote: > Ok, I think I see what you're saying: the tag only exists on the local > box and does not get forwarded in the message. You were saying you > have to overwrite the original program with some other value so that > the tag is permanent and will survive multiple relays. Sorry for the > confusion.
Try to think about the tags as a sticky note on a package. When I want to create a lot of uniform white bag I put a sticky note into it, so I can make a difference between them. But after I write the correct adresses I take off the notes from it. In the example the tag is not surviving the relays, He put the tag _value_ into an another field. Just like if I put some information into the destination address to my mail.
But I think that the idea of the persistant tags is great. And in the new syslog protocol there is space for it. I will create a feature request for it. :)
I agree. When tagging was first announced I was disappointed that they would not survive relays. Program_override is a way to 'tag' relayed messages, but it seems like a work-around (just like, say, using templates to add a custom string before $MSG to filter for at your receiving host). In a large setup, matching by host or other means is not manageable (especially if you want your configs to be multi-site compatible), so I think persistent tags would be a useful feature when sending from syslog-ng to syslog-ng.
True enough, it wasn't our original intention with the tags filter, but that's how open source works: we create a feature and you have a new idea based on that. We already had a $TAGS macro that at least makes it possible to include the set of TAGS in a message template, making it easier to transfer it to the other side. I've ported this functionality to OSE 3.1 with this patch: commit 0fed21453d7adb2a2befd534d1dea72b1d5599ee Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Apr 4 11:09:22 2010 +0200 template: added TAGS macro support which expands to a set of comma separated tag values Something like this would work for instance: rewrite r_tags { set("$TAGS" value(".SDATA.local.tags")); }; Which would include the tags value in the local.tags structured data member (when using the new syslog protocol). Of course you could also include the tags value in the message itself if you need to stick with the legacy BSD syslog. Transferring the complete message _with_ all metainformation is in our plans. (e.g. creating a serialized representation of a message and transfer that) -- Bazsi
participants (6)
-
Balazs Scheidler
-
Hoenig, Stefan, VF-Group
-
Lance Laursen
-
Martin Holste
-
SZALAY Attila
-
Zoltán Pallagi