[syslog-ng] Using source tags in rewrites?

[Robert Fekete]

Steve Barnes wrote:

>> To what end?
>> Rewrites are used to change or set key/value pairs. Flags arent key/value
>> pairs, theyre flags, meaning you either have the flag, or you dont. Think
>> of it this way; each incoming log statment has a list of flag names
>> associated with it, you can see if a certain flag name is in that list,
>> but thats it.
>> It would help to know what youre trying to accomplish though.
>> -Patrick
> 
> Patrick
> 
> I have ~70 different log files containing the results of a daily
> processing cycle. None of these log files are Syslog formatted. Unlike an
> Apache log file entry, each line of these log files is ambiguous - unless
> you referred to a portion of the file name, you'd have no way to
> differentiate a line from one file with another line from another file.
> 
> The plan is to have SyslogNG set up as a client on a machine, tracking the
> contents of these files as they are updated. Interesting lines are
> filtered on the client with matching lines pushed across the network to a
> SyslogNG server. The server is responsible for processing lines against a
> pattern DB and then stuffing the useful information into a PostgreSQL
> backend.
> 
> What I was hoping to do on the client side is tag each source in
> syslog-ng.conf with a meaningful name and then include that tag in either
> the IETF structured data or, by using a rewrite, prefix each log line
> before transmission with the tag value. This would give the SyslogNG
> server at the other end someway to differentiate messages as they arrived.
> 
> Further reading yesterday uncovered the following in the 3.2 PE
> documentation:
> 
> http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/tagging_messages.html
> 
> "To include the tags in the message, use the $TAGS macro in a template.
> Alternatively, if you are using the IETF-syslog message format, you can
> include the $TAGS macro in the .SDATA.meta part of the message. Note that
> the $TAGS macro is available only in syslog-ng PE 3.1.1 and later."
> 
> So it looks like I've found the answer to my question? Before I start
> along the path seeking purchase approval for SyslogNG PE, I wanted to make
> sure there was no other way to do what I hand in mind?


Hi Steve,

As Patrick mentioned, including the tags in the SDATA metadata should work. If 
you'd rather add the tags to the message, you can do so without rewrites if you 
use a template for the message and include the $TAGS macro in your template, but 
that is somewhat problematic because then you need to use a parser on the server 
side to separate the tags from the original message.

Using macros in rewrites should also work, though it is not explicilty mentioned 
in the docs. (I'll should add it to the next release.) Something like this 
should be equivalent to using a template:

rewrite <name_of_the_rule>
     {set("${TAGS}: ${MESSAGE}", value(MESSAGE);};

Regards,

Robert

> 
> Cheers
> 
> Steve
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 
>