[syslog-ng] Replace hostname with a hostname extracted from the message

[Clayton Dukes]

Hi Gergely,
I'm trying to understand this :-)

Where are the delimiters in the above example?
How does it know to split at a double colon?
How does it know to split at the equal?
Are they just assumed delimiters because they are non-alpha characters?

The folks I am working on this for changed the message format a bit
yesterday, so now messages appear as (example):
Nov 14 15:26:30 somehostname startofmessage:
%SYS-3-LOW_THRESHOLD:DEVICENAME Original Address=192.168.1.1
LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions

So, I was playing around with this and have the following:
filter cv_isg_events { message(".*[LOW|HIGH]_THRESHOLD.*"); };
parser p_split_at_doublecolon {
       csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy)
        );
};

parser p_split_at_equal {
       csv-parser(columns("csv.dummy", "csv.ip"),
template("${csv.orig_host}")
        );
};

rewrite r_reassemble {
        set(":${csv.message}", value("MESSAGE"));
        set("${cvs.ip}", value("HOST"));
};

destination d_tmp    {
                        file("/FNMT_SAN_DISK/fnmt/fnmt/logs/tmp_test.log");
                      };
log {
    source(net);
    filter(cv_isg_events);
    parser(p_split_at_doublecolon);
    parser(p_split_at_equal);
    rewrite(r_reassemble);
    destination(d_tmp);
};

But this doesn't yield the proper results
The IP is now missing from the host field
The DEVICENAME is missing in the message
The %SYS-3-LOW_THRESHOLD is missing

New message appearing in the tmp log:
Nov 14 15:40:41  startofmessage: :Original Address=192.168.1.1
LOW_THRESHOLD_EVENT - 0 UP sessions

Expected message (either one below is ok, probably the latter is better
since there's likely no need to maintain the original host part of the
message itself):
Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1
Original Address=192.168.1.1 LOW_THRESHOLD_EVENT - 0 UP sessions
Nov 14 15:40:41 192.168.6.3 startofmessage: %SYS-3-LOW_THRESHOLD:ISG1
LOW_THRESHOLD_EVENT - 0 UP sessions




______________________________________________________________

Clayton Dukes
______________________________________________________________


On Wed, Nov 14, 2012 at 9:57 AM, Clayton Dukes <cdukes at gmail.com> wrote:

> Awesome, thanks!
> I'll dig into the patterndb as this would have to be in production at some
> point :-)
>
>
>
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________
>
>
>
> On Wed, Nov 14, 2012 at 8:50 AM, Gergely Nagy <algernon at balabit.hu> wrote:
>
>> Clayton Dukes <cdukes at gmail.com> writes:
>>
>> > Thanks Gergely,
>> > I was trying to replace myhostname with the IP (removing the "Original
>> > Host=$IP"
>> > part of the message would be optional).
>> > I thought it would have to be done using something like patterndb, but
>> > wasn't sure.
>>
>> A quick & dirty solution is to abuse the CSV parser, twice (do note that
>> I haven't tested it):
>>
>> parser p_split_at_doublecolon {
>>        csv-parser(columns("csv.orig_host", "csv.message"), flags(greedy));
>> };
>>
>> parser p_split_at_equal {
>>        csv-parser(columns("csv.dummy", "csv.ip"),
>> template("${csv.orig_host}"));
>> };
>>
>> rewrite r_reassemble {
>>         set(":${csv.message}", value("MESSAGE"));
>>         set("${cvs.ip}", value("HOST"));
>> };
>>
>> log {
>>     source(s_yoursource);
>>     parser(p_split_at_doublecolon);
>>     parser(p_split_at_equal);
>>     rewrite(r_reassemble);
>>     destination(d_yourdestination);
>> };
>>
>> What this does, is split the original message into two parts: one before
>> the first ":", and the rest after (we need the greedy flag for this, so
>> that if the message contains more ":" chars, the parser won't split
>> there, but attach that to csv.message). The first part will be stored in
>> "csv.orig_host", the other in "csv.message", neither will contain the
>> ":" itself.
>>
>> Then, we use a similar trick to break the orig_host part apart:
>> everything in it after the equal sign is the IP.
>>
>> After this two, we have the following things set up:
>>
>> csv.orig_host = "Original Host=192.168.6.3"
>> csv.message = "LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions"
>> csv.dummy = "Original Host"
>> csv.ip = "192.168.6.3"
>>
>> From these, we can reassemble the message with a simple rewrite rule,
>> which will prepend the missing ":" to the ${csv.message}.
>>
>> This isn't the most performant thing on earth, patterndb is much faster,
>> and once you wrap your head around it, probably easier to follow
>> aswell. But this was quicker to write :P
>>
>> Hope this helps!
>>
>> --
>> |8]
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20121114/73842022/attachment-0001.htm