Balazs Scheidler wrote:
On Thu, 2007-08-16 at 10:07 -0700, Eli Stair wrote:
I've got a problem with some network devices that is leading me to need to find some way to do regex rewriting of portions of the message body of syslog messages.
Problem: I've got a bunch of Foundry devices that put their hostname followed by a comma in the body of the message (and some that do not). Some of these look like this:
{ 2007-08-16:2007-08-16T09:50:16-07:00 hostname [hostname.local7.notice] hostname, Linecard Module 13 temperature 50.0 C degrees is normal }
# template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $MSG\n")
As you can see from the template, the second hostname reported with the comma is part of the MSG body. For reasons of properly searching/indexing this data I need to strip this out. I've seen mention of a tool called 'syslog-mailer' that sounds like it would do the job somewhat. Additionally, I've seen blog chatter about potentially adding full regex rewrite capability to syslog-ng in the recent past. I can't find evidence of either of these methods however.
My first try at solving this using an external program showed that when passing data OUT of syslog-ng to a defined program, only the message body is sent and before application of a template, the other information is dropped. Thus it's not possible to do processing of the whole payload externally, re-import the data via a socket and finish writing because the facility and HOST information is all gone!
I'm looking at writing a log proxy using Net::Dev::Tools::Syslog in perl to handle listening, rewriting if necessary, and forwarding full messages on to syslog-ng after. I'd just like to know if there are any better suggestions, or if this has been done before successfully in another way?
Cheers, and thanks for any insight.
You could do something like this:
f_strip_hostname { match("^[\-a-zA-Z0-0]+,(.*)$") or match("^(.*)$"); };
destination d_out { file("/var/log/messages" template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $1\n"); };
The filter will make $1 equal to the message part without a hostname, either because there was a hostname (first match), or because there wasn't.
Improving the hostname pattern would reduce ambiguity, as the pattern will strip everything till the first comma.
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Given the example, it appears you have $1 internally hard-coded as the 1st submatch of the expressed match() rule, though I've never seen that usage/potential in the syslog-ng docs (new undocumented/testing feature?). Unfortunately I can't confirm this since it's not behaving in that manner on my system (2.0.5), hopefully you can review the result and suggest where I'm going wrong with your example? The debug output doesn't indicate whether the primary or fall-through match() is actually triggered, so that's being less than useful, and I've simplified the first match() to "^.+,(.*)$" and "^.{15}(.*)$" to force butchering the message, to see if the any $1 subclassing DOES work, and had no luck in affecting the output message line however I try. Here are two log lines for the same event on the same host, run through both the rule as suggested and an unmodifying logging rule. I bracketed $1 to distinguish the re-write rule: # log from rewrite rule as suggested: 2007-08-20T20:54:07-07:00 brr-c161-oob-01 [local7.alert] [[[brr-c161-oob-01, Power supply 1, power supply on right connector, failed ]]] # log from same template without applying match/$1 trick: 2007-08-20T20:54:07-07:00 brr-c161-oob-01 [local7.alert] brr-c161-oob-01, Power supply 1, power supply on right connector, failed ### ## Here's my rewrite ruleset as suggested by you ## (all matches behaving the same): #filter f_strip_hostname { match("^[\-a-zA-Z0-0]+,(.*)$") or match("^(.*)$"); }; #filter f_strip_hostname { match("^.+,(.*)$") or match("^(.*)$"); }; # strip everything to first comma filter f_strip_hostname { match("^.{15}(.*)$") or match("^(.*)$"); }; destination d_strip_hostname { file("/san/dept/srd/syslog/network/$HOST-test" template("$R_ISODATE $HOST [$FACILITY.$PRIORITY] [[[$1]]]\n")); }; log {source(s_net-udp); filter(f_strip_hostname); destination(d_strip_hostname); }; ## end cfg ### ## debug output showing the progression of the matches: Incoming log entry; line='<190>ldac-dc-grc-r8c15, rancid login to PRIVILEGED EXEC mode ' Filter rule evaluation begins; filter_rule='f_strip_hostname' Filter node evaluation result; filter_result='match', filter_type='match' Filter node evaluation result; filter_result='match', filter_type='OR' Filter rule evaluation result; filter_result='match', filter_rule='f_strip_hostname' Thanks for any input and suggestions on this, and everything else working so well. /eli