Replace hostname with a hostname extracted from the message
Hi Folks, How can I extract a hostname from the message and replace the source ip with that name/ip address? Sample message: Nov 13 18:11:00 myhostname mymessage: Original Host=192.168.6.3:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions So, I need a rewrite rule(?) to take 192.168.6.3 and replace $HOST with it: Nov 13 18:11:00 92.168.6.3 mymessage:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions ______________________________________________________________ Clayton Dukes ______________________________________________________________
Clayton Dukes <cdukes@gmail.com> writes:
Hi Folks, How can I extract a hostname from the message and replace the source ip with that name/ip address?
Sample message: Nov 13 18:11:00 myhostname mymessage: Original Host=192.168.6.3:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions
So, I need a rewrite rule(?) to take 192.168.6.3 and replace $HOST with it: Nov 13 18:11:00 92.168.6.3 mymessage:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions
Do you want to replace 'myhostname' with the IP, or the IP with myhostname? If you want to replace myhostname with the IP, and remove the "Original Host=$IP" part of the message, you will need to pick out the IP from the message part (either with patterndb, or some other way), then create a rewrite rule that removes it from $MESSAGE. Then, on the destination side, I'd use a template to re-assemble the thing, replacing $HOST with the extracted IP address. If you want to replace the IP with the hostname, that's a little bit easier: rewrite r_subst_ip { subst("Original Host=[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*", "$HOST", value("MESSAGE")); }; -- |8]
Thanks Gergely, I was trying to replace myhostname with the IP (removing the "Original Host=$IP" part of the message would be optional). I thought it would have to be done using something like patterndb, but wasn't sure. I'll try to read up and figure out how to do it, as my patterndb experience is a bit lacking still :-) ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Wed, Nov 14, 2012 at 5:26 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Clayton Dukes <cdukes@gmail.com> writes:
Hi Folks, How can I extract a hostname from the message and replace the source ip with that name/ip address?
Sample message: Nov 13 18:11:00 myhostname mymessage: Original Host=192.168.6.3:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions
So, I need a rewrite rule(?) to take 192.168.6.3 and replace $HOST with it: Nov 13 18:11:00 92.168.6.3 mymessage:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions
Do you want to replace 'myhostname' with the IP, or the IP with myhostname?
If you want to replace myhostname with the IP, and remove the "Original Host=$IP" part of the message, you will need to pick out the IP from the message part (either with patterndb, or some other way), then create a rewrite rule that removes it from $MESSAGE. Then, on the destination side, I'd use a template to re-assemble the thing, replacing $HOST with the extracted IP address.
If you want to replace the IP with the hostname, that's a little bit easier:
rewrite r_subst_ip { subst("Original Host=[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*", "$HOST", value("MESSAGE")); };
-- |8]
Clayton Dukes <cdukes@gmail.com> writes:
Thanks Gergely, I was trying to replace myhostname with the IP (removing the "Original Host=$IP" part of the message would be optional). I thought it would have to be done using something like patterndb, but wasn't sure.
A quick & dirty solution is to abuse the CSV parser, twice (do note that I haven't tested it): parser p_split_at_doublecolon { csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy)); }; parser p_split_at_equal { csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}")); }; rewrite r_reassemble { set(":${csv.message}", value("MESSAGE")); set("${cvs.ip}", value("HOST")); }; log { source(s_yoursource); parser(p_split_at_doublecolon); parser(p_split_at_equal); rewrite(r_reassemble); destination(d_yourdestination); }; What this does, is split the original message into two parts: one before the first ":", and the rest after (we need the greedy flag for this, so that if the message contains more ":" chars, the parser won't split there, but attach that to csv.message). The first part will be stored in "csv.orig_host", the other in "csv.message", neither will contain the ":" itself. Then, we use a similar trick to break the orig_host part apart: everything in it after the equal sign is the IP. After this two, we have the following things set up: csv.orig_host = "Original Host=192.168.6.3" csv.message = "LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions" csv.dummy = "Original Host" csv.ip = "192.168.6.3"
From these, we can reassemble the message with a simple rewrite rule, which will prepend the missing ":" to the ${csv.message}.
This isn't the most performant thing on earth, patterndb is much faster, and once you wrap your head around it, probably easier to follow aswell. But this was quicker to write :P Hope this helps! -- |8]
Awesome, thanks! I'll dig into the patterndb as this would have to be in production at some point :-) ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Wed, Nov 14, 2012 at 8:50 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Clayton Dukes <cdukes@gmail.com> writes:
Thanks Gergely, I was trying to replace myhostname with the IP (removing the "Original Host=$IP" part of the message would be optional). I thought it would have to be done using something like patterndb, but wasn't sure.
A quick & dirty solution is to abuse the CSV parser, twice (do note that I haven't tested it):
parser p_split_at_doublecolon { csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy)); };
parser p_split_at_equal { csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}")); };
rewrite r_reassemble { set(":${csv.message}", value("MESSAGE")); set("${cvs.ip}", value("HOST")); };
log { source(s_yoursource); parser(p_split_at_doublecolon); parser(p_split_at_equal); rewrite(r_reassemble); destination(d_yourdestination); };
What this does, is split the original message into two parts: one before the first ":", and the rest after (we need the greedy flag for this, so that if the message contains more ":" chars, the parser won't split there, but attach that to csv.message). The first part will be stored in "csv.orig_host", the other in "csv.message", neither will contain the ":" itself.
Then, we use a similar trick to break the orig_host part apart: everything in it after the equal sign is the IP.
After this two, we have the following things set up:
csv.orig_host = "Original Host=192.168.6.3" csv.message = "LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions" csv.dummy = "Original Host" csv.ip = "192.168.6.3"
From these, we can reassemble the message with a simple rewrite rule, which will prepend the missing ":" to the ${csv.message}.
This isn't the most performant thing on earth, patterndb is much faster, and once you wrap your head around it, probably easier to follow aswell. But this was quicker to write :P
Hope this helps!
-- |8]
Hi Gergely, I'm trying to understand this :-) Where are the delimiters in the above example? How does it know to split at a double colon? How does it know to split at the equal? Are they just assumed delimiters because they are non-alpha characters? The folks I am working on this for changed the message format a bit yesterday, so now messages appear as (example): Nov 14 15:26:30 somehostname startofmessage: %SYS-3-LOW_THRESHOLD:DEVICENAME Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions So, I was playing around with this and have the following: filter cv_isg_events { message(".*[LOW|HIGH]_THRESHOLD.*"); }; parser p_split_at_doublecolon { csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy) ); }; parser p_split_at_equal { csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}") ); }; rewrite r_reassemble { set(":${csv.message}", value("MESSAGE")); set("${cvs.ip}", value("HOST")); }; destination d_tmp { file("/FNMT_SAN_DISK/fnmt/fnmt/logs/tmp_test.log"); }; log { source(net); filter(cv_isg_events); parser(p_split_at_doublecolon); parser(p_split_at_equal); rewrite(r_reassemble); destination(d_tmp); }; But this doesn't yield the proper results The IP is now missing from the host field The DEVICENAME is missing in the message The %SYS-3-LOW_THRESHOLD is missing New message appearing in the tmp log: Nov 14 15:40:41 startofmessage: :Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 UP sessions Expected message (either one below is ok, probably the latter is better since there's likely no need to maintain the original host part of the message itself): Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1 Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 UP sessions Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1 LOW_THRESHOLD_EVENT - 0 UP sessions ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Wed, Nov 14, 2012 at 9:57 AM, Clayton Dukes <cdukes@gmail.com> wrote:
Awesome, thanks! I'll dig into the patterndb as this would have to be in production at some point :-)
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Wed, Nov 14, 2012 at 8:50 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Clayton Dukes <cdukes@gmail.com> writes:
Thanks Gergely, I was trying to replace myhostname with the IP (removing the "Original Host=$IP" part of the message would be optional). I thought it would have to be done using something like patterndb, but wasn't sure.
A quick & dirty solution is to abuse the CSV parser, twice (do note that I haven't tested it):
parser p_split_at_doublecolon { csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy)); };
parser p_split_at_equal { csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}")); };
rewrite r_reassemble { set(":${csv.message}", value("MESSAGE")); set("${cvs.ip}", value("HOST")); };
log { source(s_yoursource); parser(p_split_at_doublecolon); parser(p_split_at_equal); rewrite(r_reassemble); destination(d_yourdestination); };
What this does, is split the original message into two parts: one before the first ":", and the rest after (we need the greedy flag for this, so that if the message contains more ":" chars, the parser won't split there, but attach that to csv.message). The first part will be stored in "csv.orig_host", the other in "csv.message", neither will contain the ":" itself.
Then, we use a similar trick to break the orig_host part apart: everything in it after the equal sign is the IP.
After this two, we have the following things set up:
csv.orig_host = "Original Host=192.168.6.3" csv.message = "LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions" csv.dummy = "Original Host" csv.ip = "192.168.6.3"
From these, we can reassemble the message with a simple rewrite rule, which will prepend the missing ":" to the ${csv.message}.
This isn't the most performant thing on earth, patterndb is much faster, and once you wrap your head around it, probably easier to follow aswell. But this was quicker to write :P
Hope this helps!
-- |8]
Clayton Dukes <cdukes@gmail.com> writes:
Hi Gergely, I'm trying to understand this :-)
Where are the delimiters in the above example? How does it know to split at a double colon? How does it know to split at the equal?
Duh! I forgot to include them (so it defaults to whitespace). Add a delimiters(":") to the first, and delimiters("=") to the second. Sorry!
Are they just assumed delimiters because they are non-alpha characters?
The folks I am working on this for changed the message format a bit yesterday, so now messages appear as (example): Nov 14 15:26:30 somehostname startofmessage: %SYS-3-LOW_THRESHOLD:DEVICENAME Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions
This'll be a little harder to do with csv-parser, but not impossible! You'd need to split on "=" first (into, say, csv.first_part and csv.second_part). Then split the first part on whitespace (into csv.cisco_thingy, and some dummy), then split the second part on whitespace (into csv.ip and csv.message). -- |8]
participants (2)
-
Clayton Dukes
-
Gergely Nagy