Hi Gergely,
I'm trying to understand this :-)

Where are the delimiters in the above example?
How does it know to split at a double colon?
How does it know to split at the equal?
Are they just assumed delimiters because they are non-alpha characters?

The folks I am working on this for changed the message format a bit yesterday, so now messages appear as (example):
Nov 14 15:26:30 somehostname startofmessage: %SYS-3-LOW_THRESHOLD:DEVICENAME Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions

So, I was playing around with this and have the following:
filter cv_isg_events { message(".*[LOW|HIGH]_THRESHOLD.*"); };
parser p_split_at_doublecolon {
       csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy)
        );
};

parser p_split_at_equal {
       csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}")
        );
};

rewrite r_reassemble {
        set(":${csv.message}", value("MESSAGE"));
        set("${cvs.ip}", value("HOST"));
};

destination d_tmp    {
                        file("/FNMT_SAN_DISK/fnmt/fnmt/logs/tmp_test.log");
                      };
log {
    source(net);
    filter(cv_isg_events);
    parser(p_split_at_doublecolon);
    parser(p_split_at_equal);
    rewrite(r_reassemble);
    destination(d_tmp);
};

But this doesn't yield the proper results
The IP is now missing from the host field
The DEVICENAME is missing in the message
The %SYS-3-LOW_THRESHOLD is missing

New message appearing in the tmp log:
Nov 14 15:40:41  startofmessage: :Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 UP sessions

Expected message (either one below is ok, probably the latter is better since there's likely no need to maintain the original host part of the message itself):
Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1 Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 UP sessions
Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1 LOW_THRESHOLD_EVENT - 0 UP sessions




______________________________________________________________

Clayton Dukes
______________________________________________________________


On Wed, Nov 14, 2012 at 9:57 AM, Clayton Dukes <cdukes@gmail.com> wrote:
Awesome, thanks!
I'll dig into the patterndb as this would have to be in production at some point :-)



______________________________________________________________

Clayton Dukes
______________________________________________________________



On Wed, Nov 14, 2012 at 8:50 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Clayton Dukes <cdukes@gmail.com> writes:

> Thanks Gergely,
> I was trying to replace myhostname with the IP (removing the "Original
> Host=$IP"
> part of the message would be optional).
> I thought it would have to be done using something like patterndb, but
> wasn't sure.

A quick & dirty solution is to abuse the CSV parser, twice (do note that
I haven't tested it):

parser p_split_at_doublecolon {
       csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy));
};

parser p_split_at_equal {
       csv-parser(columns("csv.dummy", "csv.ip"), template("${csv.orig_host}"));
};

rewrite r_reassemble {
        set(":${csv.message}", value("MESSAGE"));
        set("${cvs.ip}", value("HOST"));
};

log {
    source(s_yoursource);
    parser(p_split_at_doublecolon);
    parser(p_split_at_equal);
    rewrite(r_reassemble);
    destination(d_yourdestination);
};

What this does, is split the original message into two parts: one before
the first ":", and the rest after (we need the greedy flag for this, so
that if the message contains more ":" chars, the parser won't split
there, but attach that to csv.message). The first part will be stored in
"csv.orig_host", the other in "csv.message", neither will contain the
":" itself.

Then, we use a similar trick to break the orig_host part apart:
everything in it after the equal sign is the IP.

After this two, we have the following things set up:

csv.orig_host = "Original Host=192.168.6.3"
csv.message = "LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions"
csv.dummy = "Original Host"
csv.ip = "192.168.6.3"

>From these, we can reassemble the message with a simple rewrite rule,
which will prepend the missing ":" to the ${csv.message}.

This isn't the most performant thing on earth, patterndb is much faster,
and once you wrap your head around it, probably easier to follow
aswell. But this was quicker to write :P

Hope this helps!

--
|8]