[syslog-ng] REGEX rewrites on packet body possible?

[Balazs Scheidler]

On Thu, 2007-08-16 at 10:07 -0700, Eli Stair wrote:
> I've got a problem with some network devices that is leading me to need to find 
> some way to do regex rewriting of portions of the message body of syslog 
> messages.
> 
> Problem: I've got a bunch of Foundry devices that put their hostname followed 
> by a comma in the body of the message (and some that do not).  Some of these 
> look like this:
> 
> {
> 2007-08-16:2007-08-16T09:50:16-07:00 hostname [hostname.local7.notice] 
> hostname, Linecard Module 13 temperature 50.0 C degrees is normal
> }
> 
> # template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $MSG\n")
> 
> 
> As you can see from the template, the second hostname reported with the comma 
> is part of the MSG body.  For reasons of properly searching/indexing this data 
> I need to strip this out.  I've seen mention of a tool called 'syslog-mailer' 
> that sounds like it would do the job somewhat.  Additionally, I've seen blog 
> chatter about potentially adding full regex rewrite capability to syslog-ng in 
> the recent past.  I can't find evidence of either of these methods however.
> 
> My first try at solving this using an external program showed that when passing 
> data OUT of syslog-ng to a defined program, only the message body is sent and 
> before application of a template, the other information is dropped.  Thus it's 
> not possible to do processing of the whole payload externally, re-import the 
> data via a socket and finish writing because the facility and HOST information 
> is all gone!
> 
> I'm looking at writing a log proxy using Net::Dev::Tools::Syslog in perl to 
> handle listening, rewriting if necessary, and forwarding full messages on to 
> syslog-ng after.  I'd just like to know if there are any better suggestions, or 
> if this has been done before successfully in another way?
> 
> Cheers, and thanks for any insight.

You could do something like this:

f_strip_hostname { match("^[\-a-zA-Z0-0]+,(.*)$") or match("^(.*)$"); };

destination d_out { file("/var/log/messages" template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $1\n"); };

The filter will make $1 equal to the message part without a hostname, either because 
there was a hostname (first match), or because there wasn't.

Improving the hostname pattern would reduce ambiguity, as the pattern will strip 
everything till the first comma.

-- 
Bazsi