[syslog-ng] Using source tags in rewrites?

[Balazs Scheidler]

On Wed, 2010-08-11 at 09:12 +1000, Steve Barnes wrote:
> > To what end?
> > Rewrites are used to change or set key/value pairs. Flags arent key/value
> > pairs, theyre flags, meaning you either have the flag, or you dont. Think
> > of it this way; each incoming log statment has a list of flag names
> > associated with it, you can see if a certain flag name is in that list,
> > but thats it.
> > It would help to know what youre trying to accomplish though.
> > -Patrick
> 
> Patrick
> 
> I have ~70 different log files containing the results of a daily
> processing cycle. None of these log files are Syslog formatted. Unlike an
> Apache log file entry, each line of these log files is ambiguous - unless
> you referred to a portion of the file name, you'd have no way to
> differentiate a line from one file with another line from another file.
> 
> The plan is to have SyslogNG set up as a client on a machine, tracking the
> contents of these files as they are updated. Interesting lines are
> filtered on the client with matching lines pushed across the network to a
> SyslogNG server. The server is responsible for processing lines against a
> pattern DB and then stuffing the useful information into a PostgreSQL
> backend.
> 
> What I was hoping to do on the client side is tag each source in
> syslog-ng.conf with a meaningful name and then include that tag in either
> the IETF structured data or, by using a rewrite, prefix each log line
> before transmission with the tag value. This would give the SyslogNG
> server at the other end someway to differentiate messages as they arrived.
> 
> Further reading yesterday uncovered the following in the 3.2 PE
> documentation:
> 
> http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging_messages.html
> 
> "To include the tags in the message, use the $TAGS macro in a template.
> Alternatively, if you are using the IETF-syslog message format, you can
> include the $TAGS macro in the .SDATA.meta part of the message. Note that
> the $TAGS macro is available only in syslog-ng PE 3.1.1 and later."

You were probably looking at the PE docs, but OSE - of course - also has
this macro in 3.1.2

> 
> So it looks like I've found the answer to my question? Before I start
> along the path seeking purchase approval for SyslogNG PE, I wanted to make
> sure there was no other way to do what I hand in mind?

It'd be mean to require a PE license just for a "$TAGS" macro. :) We are
not that evil.

You could perhaps also apply this macro to a structured data member, if
you don't want to touch the message payload.

For example on the client:

rewrite s_tags { set("$TAGS" value(".SDATA.meta@<enterprise
OID>.tags")); };

This should result in a .SDATA.meta@<enterpriseOID>.tags name-value pair
on the server side. Then you could either use it as is, or possibly
process it further using csv-parser:

parser p_tags { csv-parser(separator(',') template("${.SDATA.meta@<enterprise OID>.tags}")); };

The <enterpriseOID> stuff is optional, basically it ensures that locally 
assigned structured data members do not conflict. Read the 
RFC5424 for more information. (you could of course omit that but risk that 
other RFC5424 application would start using the ID you picked)

-- 
Bazsi