[syslog-ng] REGEX rewrites on packet body possible?

[Eli Stair]

Balazs Scheidler wrote:
> On Thu, 2007-08-16 at 10:07 -0700, Eli Stair wrote:
>  > I've got a problem with some network devices that is leading me to 
> need to find
>  > some way to do regex rewriting of portions of the message body of syslog
>  > messages.
>  >
>  > Problem: I've got a bunch of Foundry devices that put their hostname 
> followed
>  > by a comma in the body of the message (and some that do not).  Some 
> of these
>  > look like this:
>  >
>  > {
>  > 2007-08-16:2007-08-16T09:50:16-07:00 hostname [hostname.local7.notice]
>  > hostname, Linecard Module 13 temperature 50.0 C degrees is normal
>  > }
>  >
>  > # template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $MSG\n")
>  >
>  >
>  > As you can see from the template, the second hostname reported with 
> the comma
>  > is part of the MSG body.  For reasons of properly searching/indexing 
> this data
>  > I need to strip this out.  I've seen mention of a tool called 
> 'syslog-mailer'
>  > that sounds like it would do the job somewhat.  Additionally, I've 
> seen blog
>  > chatter about potentially adding full regex rewrite capability to 
> syslog-ng in
>  > the recent past.  I can't find evidence of either of these methods 
> however.
>  >
>  > My first try at solving this using an external program showed that 
> when passing
>  > data OUT of syslog-ng to a defined program, only the message body is 
> sent and
>  > before application of a template, the other information is dropped.  
> Thus it's
>  > not possible to do processing of the whole payload externally, 
> re-import the
>  > data via a socket and finish writing because the facility and HOST 
> information
>  > is all gone!
>  >
>  > I'm looking at writing a log proxy using Net::Dev::Tools::Syslog in 
> perl to
>  > handle listening, rewriting if necessary, and forwarding full 
> messages on to
>  > syslog-ng after.  I'd just like to know if there are any better 
> suggestions, or
>  > if this has been done before successfully in another way?
>  >
>  > Cheers, and thanks for any insight.
> 
> You could do something like this:
> 
> f_strip_hostname { match("^[\-a-zA-Z0-0]+,(.*)$") or match("^(.*)$"); };
> 
> destination d_out { file("/var/log/messages" template("$R_ISODATE $HOST 
> [$PROGRAM.$FACILITY.$PRIORITY] $1\n"); };
> 
> The filter will make $1 equal to the message part without a hostname, 
> either because
> there was a hostname (first match), or because there wasn't.
> 
> Improving the hostname pattern would reduce ambiguity, as the pattern 
> will strip
> everything till the first comma.
> 
> --
> Bazsi
> 
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
> 


Given the example, it appears you have $1 internally hard-coded as the 1st 
submatch of the expressed match() rule, though I've never seen that 
usage/potential in the syslog-ng docs (new undocumented/testing feature?). 
Unfortunately I can't confirm this since it's not behaving in that manner on my 
system (2.0.5), hopefully you can review the result and suggest where I'm going 
wrong with your example?

The debug output doesn't indicate whether the primary or fall-through match() 
is actually triggered, so that's being less than useful, and I've simplified 
the first match() to "^.+,(.*)$" and "^.{15}(.*)$" to force butchering the 
message, to see if the any $1 subclassing DOES work, and had no luck in 
affecting the output message line however I try.

Here are two log lines for the same event on the same host, run through both 
the rule as suggested and an unmodifying logging rule.  I bracketed $1 to 
distinguish the re-write rule:

# log from rewrite rule as suggested:
2007-08-20T20:54:07-07:00 brr-c161-oob-01 [local7.alert] [[[brr-c161-oob-01, 
Power supply 1, power supply on right connector, failed ]]]

# log from same template without applying match/$1 trick:
2007-08-20T20:54:07-07:00 brr-c161-oob-01 [local7.alert] brr-c161-oob-01, Power 
supply 1, power supply on right connector, failed

###
## Here's my rewrite ruleset as suggested by you
## (all matches behaving the same):
#filter f_strip_hostname { match("^[\-a-zA-Z0-0]+,(.*)$") or match("^(.*)$"); };
#filter f_strip_hostname { match("^.+,(.*)$") or match("^(.*)$"); }; # strip 
everything to first comma

filter f_strip_hostname { match("^.{15}(.*)$") or match("^(.*)$"); };
destination d_strip_hostname {
   file("/san/dept/srd/syslog/network/$HOST-test"
   template("$R_ISODATE $HOST [$FACILITY.$PRIORITY] [[[$1]]]\n"));
};
log {source(s_net-udp); filter(f_strip_hostname); destination(d_strip_hostname); };

## end cfg


###
## debug output showing the progression of the matches:
Incoming log entry; line='<190>ldac-dc-grc-r8c15, rancid login to PRIVILEGED 
EXEC mode '
Filter rule evaluation begins; filter_rule='f_strip_hostname'
Filter node evaluation result; filter_result='match', filter_type='match'
Filter node evaluation result; filter_result='match', filter_type='OR'
Filter rule evaluation result; filter_result='match', 
filter_rule='f_strip_hostname'


Thanks for any input and suggestions on this, and everything else working so well.


/eli