Hi Gergely,<div>I'm trying to understand this :-)<br></div><div><br></div><div>Where are the delimiters in the above example?</div><div>How does it know to split at a double colon?</div><div>How does it know to split at the equal?</div>
<div>Are they just assumed delimiters because they are non-alpha characters?</div><div><br></div><div>The folks I am working on this for changed the message format a bit yesterday, so now messages appear as (example):</div>
<div>Nov 14 15:26:30 somehostname startofmessage: %SYS-3-LOW_THRESHOLD:DEVICENAME Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions<br></div><div><br></div><div>So, I was playing around with this and have the following:</div>
<div>filter cv_isg_events { message(".*[LOW|HIGH]_THRESHOLD.*"); };<br></div><div><div>parser p_split_at_doublecolon {</div><div> csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy)</div>
<div> );</div><div>};</div><div><br></div><div>parser p_split_at_equal {</div><div> csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}")</div><div> );</div>
<div>};</div><div><br></div><div>rewrite r_reassemble {</div><div> set(":${csv.message}", value("MESSAGE"));</div><div> set("${cvs.ip}", value("HOST"));</div><div>};</div>
<div><br></div><div>destination d_tmp {</div><div> file("/FNMT_SAN_DISK/fnmt/fnmt/logs/tmp_test.log");</div><div> };</div><div>log {</div><div> source(net);</div>
<div> filter(cv_isg_events);</div><div> parser(p_split_at_doublecolon);</div><div> parser(p_split_at_equal);</div><div> rewrite(r_reassemble);</div><div> destination(d_tmp);<br></div><div>};</div></div><div>
<br></div><div>But this doesn't yield the proper results</div><div>The IP is now missing from the host field</div><div>The DEVICENAME is missing in the message</div><div>The %SYS-3-LOW_THRESHOLD is missing</div><div><br>
</div><div>New message appearing in the tmp log:</div><div>Nov 14 15:40:41 startofmessage: :Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 UP sessions<br></div><div><br></div><div>Expected message (either one below is ok, probably the latter is better since there's likely no need to maintain the original host part of the message itself):</div>
<div>Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1 Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 UP sessions<br></div><div>Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1 LOW_THRESHOLD_EVENT - 0 UP sessions<br>
</div><div><br></div><div><br></div><div><br></div><div class="gmail_extra"><br clear="all">______________________________________________________________ <br><br>Clayton Dukes<br>______________________________________________________________<br>
<br><br><div class="gmail_quote">On Wed, Nov 14, 2012 at 9:57 AM, Clayton Dukes <span dir="ltr"><<a href="mailto:cdukes@gmail.com" target="_blank">cdukes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Awesome, thanks!<div>I'll dig into the patterndb as this would have to be in production at some point :-)</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br></div></font></span><div class="gmail_extra">
<span class="HOEnZb"><font color="#888888"><br clear="all">______________________________________________________________ <br>
<br>Clayton Dukes<br>______________________________________________________________</font></span><div><div class="h5"><br>
<br><br><div class="gmail_quote">On Wed, Nov 14, 2012 at 8:50 AM, Gergely Nagy <span dir="ltr"><<a href="mailto:algernon@balabit.hu" target="_blank">algernon@balabit.hu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Clayton Dukes <<a href="mailto:cdukes@gmail.com" target="_blank">cdukes@gmail.com</a>> writes:<br>
<br>
> Thanks Gergely,<br>
> I was trying to replace myhostname with the IP (removing the "Original<br>
> Host=$IP"<br>
> part of the message would be optional).<br>
> I thought it would have to be done using something like patterndb, but<br>
> wasn't sure.<br>
<br>
</div>A quick & dirty solution is to abuse the CSV parser, twice (do note that<br>
I haven't tested it):<br>
<br>
parser p_split_at_doublecolon {<br>
csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy));<br>
};<br>
<br>
parser p_split_at_equal {<br>
csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}"));<br>
};<br>
<br>
rewrite r_reassemble {<br>
set(":${csv.message}", value("MESSAGE"));<br>
set("${cvs.ip}", value("HOST"));<br>
};<br>
<br>
log {<br>
source(s_yoursource);<br>
parser(p_split_at_doublecolon);<br>
parser(p_split_at_equal);<br>
rewrite(r_reassemble);<br>
destination(d_yourdestination);<br>
};<br>
<br>
What this does, is split the original message into two parts: one before<br>
the first ":", and the rest after (we need the greedy flag for this, so<br>
that if the message contains more ":" chars, the parser won't split<br>
there, but attach that to csv.message). The first part will be stored in<br>
"csv.orig_host", the other in "csv.message", neither will contain the<br>
":" itself.<br>
<br>
Then, we use a similar trick to break the orig_host part apart:<br>
everything in it after the equal sign is the IP.<br>
<br>
After this two, we have the following things set up:<br>
<br>
csv.orig_host = "Original Host=192.168.6.3"<br>
csv.message = "LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions"<br>
csv.dummy = "Original Host"<br>
csv.ip = "192.168.6.3"<br>
<br>