Using source tags in rewrites?
Hi Is it possible to use source tags in rewrite rules? The documentation indicates only macros are acceptable? Using Syslog 3.1.2. Cheers Steve
To what end? Rewrites are used to change or set key/value pairs. Flags arent key/value pairs, theyre flags, meaning you either have the flag, or you dont. Think of it this way; each incoming log statment has a list of flag names associated with it, you can see if a certain flag name is in that list, but thats it. It would help to know what youre trying to accomplish though. -Patrick Sent: Monday, August 09, 2010 10:29:10 PM From: Steve Barnes <steve@echo.id.au> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] Using source tags in rewrites?
Hi
Is it possible to use source tags in rewrite rules? The documentation indicates only macros are acceptable?
Using Syslog 3.1.2.
Cheers
Steve
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
To what end? Rewrites are used to change or set key/value pairs. Flags arent key/value pairs, theyre flags, meaning you either have the flag, or you dont. Think of it this way; each incoming log statment has a list of flag names associated with it, you can see if a certain flag name is in that list, but thats it. It would help to know what youre trying to accomplish though. -Patrick
Patrick I have ~70 different log files containing the results of a daily processing cycle. None of these log files are Syslog formatted. Unlike an Apache log file entry, each line of these log files is ambiguous - unless you referred to a portion of the file name, you'd have no way to differentiate a line from one file with another line from another file. The plan is to have SyslogNG set up as a client on a machine, tracking the contents of these files as they are updated. Interesting lines are filtered on the client with matching lines pushed across the network to a SyslogNG server. The server is responsible for processing lines against a pattern DB and then stuffing the useful information into a PostgreSQL backend. What I was hoping to do on the client side is tag each source in syslog-ng.conf with a meaningful name and then include that tag in either the IETF structured data or, by using a rewrite, prefix each log line before transmission with the tag value. This would give the SyslogNG server at the other end someway to differentiate messages as they arrived. Further reading yesterday uncovered the following in the 3.2 PE documentation: http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging... "To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng PE 3.1.1 and later." So it looks like I've found the answer to my question? Before I start along the path seeking purchase approval for SyslogNG PE, I wanted to make sure there was no other way to do what I hand in mind? Cheers Steve
Ok, I see what youre doing now :-) There's another simple solution that pops into mind; the file() source driver supports the 'program-override()' option. Basically any logs read from that file would have whatever $PROGRAM you specified. And that $PROGRAM macro will be propagated from the client syslog-ng to your server. -Patrick Sent: Tuesday, August 10, 2010 5:12:10 PM From: Steve Barnes <steve@echo.id.au> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Using source tags in rewrites?
To what end? Rewrites are used to change or set key/value pairs. Flags arent key/value pairs, theyre flags, meaning you either have the flag, or you dont. Think of it this way; each incoming log statment has a list of flag names associated with it, you can see if a certain flag name is in that list, but thats it. It would help to know what youre trying to accomplish though. -Patrick
Patrick
I have ~70 different log files containing the results of a daily processing cycle. None of these log files are Syslog formatted. Unlike an Apache log file entry, each line of these log files is ambiguous - unless you referred to a portion of the file name, you'd have no way to differentiate a line from one file with another line from another file.
The plan is to have SyslogNG set up as a client on a machine, tracking the contents of these files as they are updated. Interesting lines are filtered on the client with matching lines pushed across the network to a SyslogNG server. The server is responsible for processing lines against a pattern DB and then stuffing the useful information into a PostgreSQL backend.
What I was hoping to do on the client side is tag each source in syslog-ng.conf with a meaningful name and then include that tag in either the IETF structured data or, by using a rewrite, prefix each log line before transmission with the tag value. This would give the SyslogNG server at the other end someway to differentiate messages as they arrived.
Further reading yesterday uncovered the following in the 3.2 PE documentation:
http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging...
"To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng PE 3.1.1 and later."
So it looks like I've found the answer to my question? Before I start along the path seeking purchase approval for SyslogNG PE, I wanted to make sure there was no other way to do what I hand in mind?
Cheers
Steve
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Patrick I did consider the program-override together with the $PROGRAM macro, but I was concerned that ~70 different $PROGRAM types would come back to haunt me some where down the line. Having a single $PROGRAM to summarise the log file related traffic felt like a less "risky" / more manageable approach. Saying that, it looks like I don't have much choice given that I'm presently using the OSE version. Thanks for the suggestion, I'll give it a whirl and see what the end result looks like :-) Cheers Steve
Ok, I see what youre doing now :-) There's another simple solution that pops into mind; the file() source driver supports the 'program-override()' option. Basically any logs read from that file would have whatever $PROGRAM you specified. And that $PROGRAM macro will be propagated from the client syslog-ng to your server. -Patrick Sent: Tuesday, August 10, 2010 5:12:10 PM From: Steve Barnes mailto:steve@echo.id.au <steve@echo.id.au> To: Syslog-ng users' and developers' mailing list mailto:syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Using source tags in rewrites? To what end? Rewrites are used to change or set key/value pairs. Flags arent key/value pairs, theyre flags, meaning you either have the flag, or you dont. Think of it this way; each incoming log statment has a list of flag names associated with it, you can see if a certain flag name is in that list, but thats it. It would help to know what youre trying to accomplish though. -Patrick Patrick I have ~70 different log files containing the results of a daily processing cycle. None of these log files are Syslog formatted. Unlike an Apache log file entry, each line of these log files is ambiguous - unless you referred to a portion of the file name, you'd have no way to differentiate a line from one file with another line from another file. The plan is to have SyslogNG set up as a client on a machine, tracking the contents of these files as they are updated. Interesting lines are filtered on the client with matching lines pushed across the network to a SyslogNG server. The server is responsible for processing lines against a pattern DB and then stuffing the useful information into a PostgreSQL backend. What I was hoping to do on the client side is tag each source in syslog-ng.conf with a meaningful name and then include that tag in either the IETF structured data or, by using a rewrite, prefix each log line before transmission with the tag value. This would give the SyslogNG server at the other end someway to differentiate messages as they arrived. Further reading yesterday uncovered the following in the 3.2 PE documentation: http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging... http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging... "To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng PE 3.1.1 and later." So it looks like I've found the answer to my question? Before I start along the path seeking purchase approval for SyslogNG PE, I wanted to make sure there was no other way to do what I hand in mind? Cheers Steve ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html http://www.campin.net/syslog-ng/faq.html ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
You can do something like "program-override('logfile/category/subcat')". that way later down the line, if you need to test for entries coming from these sources, you can do "program('logfile')" inside a filter, or "program('logfile/category')" to match a certain category of the log files or whatnot. -Patrick Sent: Wednesday, August 11, 2010 7:34:16 AM From: Steve Barnes <steve@echo.id.au> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Using source tags in rewrites?
Patrick
I did consider the program-override together with the $PROGRAM macro, but I was concerned that ~70 different $PROGRAM types would come back to haunt me some where down the line. Having a single $PROGRAM to summarise the log file related traffic felt like a less "risky" / more manageable approach. Saying that, it looks like I don't have much choice given that I'm presently using the OSE version.
Thanks for the suggestion, I'll give it a whirl and see what the end result looks like :-)
Cheers
Steve
Ok, I see what youre doing now :-) There's another simple solution that pops into mind; the file() source driver supports the 'program-override()' option. Basically any logs read from that file would have whatever $PROGRAM you specified. And that $PROGRAM macro will be propagated from the client syslog-ng to your server. -Patrick Sent: Tuesday, August 10, 2010 5:12:10 PM From: Steve Barnes mailto:steve@echo.id.au <steve@echo.id.au> To: Syslog-ng users' and developers' mailing list mailto:syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Using source tags in rewrites? To what end? Rewrites are used to change or set key/value pairs. Flags arent key/value pairs, theyre flags, meaning you either have the flag, or you dont. Think of it this way; each incoming log statment has a list of flag names associated with it, you can see if a certain flag name is in that list, but thats it. It would help to know what youre trying to accomplish though. -Patrick Patrick I have ~70 different log files containing the results of a daily processing cycle. None of these log files are Syslog formatted. Unlike an Apache log file entry, each line of these log files is ambiguous - unless you referred to a portion of the file name, you'd have no way to differentiate a line from one file with another line from another file. The plan is to have SyslogNG set up as a client on a machine, tracking the contents of these files as they are updated. Interesting lines are filtered on the client with matching lines pushed across the network to a SyslogNG server. The server is responsible for processing lines against a pattern DB and then stuffing the useful information into a PostgreSQL backend. What I was hoping to do on the client side is tag each source in syslog-ng.conf with a meaningful name and then include that tag in either the IETF structured data or, by using a rewrite, prefix each log line before transmission with the tag value. This would give the SyslogNG server at the other end someway to differentiate messages as they arrived. Further reading yesterday uncovered the following in the 3.2 PE documentation: http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging... http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging... "To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng PE 3.1.1 and later." So it looks like I've found the answer to my question? Before I start along the path seeking purchase approval for SyslogNG PE, I wanted to make sure there was no other way to do what I hand in mind? Cheers Steve ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html http://www.campin.net/syslog-ng/faq.html ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Steve Barnes wrote:
To what end? Rewrites are used to change or set key/value pairs. Flags arent key/value pairs, theyre flags, meaning you either have the flag, or you dont. Think of it this way; each incoming log statment has a list of flag names associated with it, you can see if a certain flag name is in that list, but thats it. It would help to know what youre trying to accomplish though. -Patrick
Patrick
I have ~70 different log files containing the results of a daily processing cycle. None of these log files are Syslog formatted. Unlike an Apache log file entry, each line of these log files is ambiguous - unless you referred to a portion of the file name, you'd have no way to differentiate a line from one file with another line from another file.
The plan is to have SyslogNG set up as a client on a machine, tracking the contents of these files as they are updated. Interesting lines are filtered on the client with matching lines pushed across the network to a SyslogNG server. The server is responsible for processing lines against a pattern DB and then stuffing the useful information into a PostgreSQL backend.
What I was hoping to do on the client side is tag each source in syslog-ng.conf with a meaningful name and then include that tag in either the IETF structured data or, by using a rewrite, prefix each log line before transmission with the tag value. This would give the SyslogNG server at the other end someway to differentiate messages as they arrived.
Further reading yesterday uncovered the following in the 3.2 PE documentation:
http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging...
"To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng PE 3.1.1 and later."
So it looks like I've found the answer to my question? Before I start along the path seeking purchase approval for SyslogNG PE, I wanted to make sure there was no other way to do what I hand in mind?
Hi Steve, As Patrick mentioned, including the tags in the SDATA metadata should work. If you'd rather add the tags to the message, you can do so without rewrites if you use a template for the message and include the $TAGS macro in your template, but that is somewhat problematic because then you need to use a parser on the server side to separate the tags from the original message. Using macros in rewrites should also work, though it is not explicilty mentioned in the docs. (I'll should add it to the next release.) Something like this should be equivalent to using a template: rewrite <name_of_the_rule> {set("${TAGS}: ${MESSAGE}", value(MESSAGE);}; Regards, Robert
Cheers
Steve
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2010-08-11 at 09:12 +1000, Steve Barnes wrote:
To what end? Rewrites are used to change or set key/value pairs. Flags arent key/value pairs, theyre flags, meaning you either have the flag, or you dont. Think of it this way; each incoming log statment has a list of flag names associated with it, you can see if a certain flag name is in that list, but thats it. It would help to know what youre trying to accomplish though. -Patrick
Patrick
I have ~70 different log files containing the results of a daily processing cycle. None of these log files are Syslog formatted. Unlike an Apache log file entry, each line of these log files is ambiguous - unless you referred to a portion of the file name, you'd have no way to differentiate a line from one file with another line from another file.
The plan is to have SyslogNG set up as a client on a machine, tracking the contents of these files as they are updated. Interesting lines are filtered on the client with matching lines pushed across the network to a SyslogNG server. The server is responsible for processing lines against a pattern DB and then stuffing the useful information into a PostgreSQL backend.
What I was hoping to do on the client side is tag each source in syslog-ng.conf with a meaningful name and then include that tag in either the IETF structured data or, by using a rewrite, prefix each log line before transmission with the tag value. This would give the SyslogNG server at the other end someway to differentiate messages as they arrived.
Further reading yesterday uncovered the following in the 3.2 PE documentation:
http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging...
"To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng PE 3.1.1 and later."
You were probably looking at the PE docs, but OSE - of course - also has this macro in 3.1.2
So it looks like I've found the answer to my question? Before I start along the path seeking purchase approval for SyslogNG PE, I wanted to make sure there was no other way to do what I hand in mind?
It'd be mean to require a PE license just for a "$TAGS" macro. :) We are not that evil. You could perhaps also apply this macro to a structured data member, if you don't want to touch the message payload. For example on the client: rewrite s_tags { set("$TAGS" value(".SDATA.meta@<enterprise OID>.tags")); }; This should result in a .SDATA.meta@<enterpriseOID>.tags name-value pair on the server side. Then you could either use it as is, or possibly process it further using csv-parser: parser p_tags { csv-parser(separator(',') template("${.SDATA.meta@<enterprise OID>.tags}")); }; The <enterpriseOID> stuff is optional, basically it ensures that locally assigned structured data members do not conflict. Read the RFC5424 for more information. (you could of course omit that but risk that other RFC5424 application would start using the ID you picked) -- Bazsi
participants (4)
-
Balazs Scheidler
-
Robert Fekete
-
Steve Barnes
-
syslogng＠feystorm.net