[syslog-ng] Rewriting Cisco messages

[Thomas Wollner]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Martin,

mine is a simplified one of yours (and yes stolen / inspired from elsa;-))

filter f_rewrite_cisco_program {
  match('%([^:]+):\s+([^\n]+)' value("MESSAGE") type("pcre")
flags("store-matches" "nobackref"));
};

rewrite r_cisco_program {
  set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program)));
  set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program)));
};

just one step and in the rewrite everything before the % gets whiped out.

I`m pre-filtering for cisco based logs (which have the mnemonic in the
message) and then rewrite only those. I`have logs from different
network vendors and some servers too. The combined pre-filter together
with the rewrite I newer catched a "false" one. Currently I have
around 6-8 k msg/sec and I do not notice an impact on cpu utilization
compared to the setup without the rewriting.


best regards,

Tom




On 07.02.2012 22:45, Martin Holste wrote:
> Yep, my regexes should account for all of those weird formats.
> So, your method is a two-step method in which you first clear out 
> everything before the percent sign, then you match from the percent
> to the colon?  Are you only receiving Cisco syslog?  Do you have a
> config example?
> 
> On Tue, Feb 7, 2012 at 2:16 PM, Thomas Wollner <tw at wollner-net.de>
> wrote:
> 
> 
> Martin,
> 
> the configs on cisco routers are very different. One can use too
> many options how timestamps are built and that crap makes it to the
> message part of the log.
> 
> Just an example: #service timestamps log myrouter(config)#service
> timestamps log ? datetime  Timestamp with date and time uptime
> Timestamp with system uptime <cr>
> 
> myrouter(config)#service timestamps log date 
> myrouter(config)#service timestamps log datetime ? localtime
> Use local time zone for timestamps msec           Include
> milliseconds in timestamp show-timezone  Add time zone information
> to timestamp year           Include year in timestamp <cr>
> 
> Because of the too many options I used to rewrite all that crap
> with nothing until I find a %FACILITY-PRIORITY-PROGRAM - a so
> called mnemonic - and set that as the $PROGRAM without leading %.
> That way I can use a simple regex "everything between % and :" is
> program and everything until end of line is message.
> 
> The above options are just for ios based devices. ios-xe, ios-xr, 
> nx-os are out in the wild now and I`m quite shure they will
> operate different...
> 
> just my 2 cents,
> 
> best regards,
> 
> Tom
> 
> 
> 
> On 07.02.2012 16:26, Martin Holste wrote:
>>>> I spent some time yesterday working out the proper regexes
>>>> to handle the many ways Cisco sends its timestamps, depending
>>>> on how the device is configured.  However, I feel like my
>>>> solution can be improved upon, so I'd like to see if there's
>>>> a better way.  Here's what I've got so far that seems to be
>>>> working, though there seems to be a 10-15% CPU penalty at the
>>>> moment:
>>>> 
>>>> #4w4d: %LINEPROTO-5-UPDOWN: Line protocol on Interface 
>>>> FastEthernet0/15, changed state to down #1y46w: 
>>>> %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered
>>>> on GigabitEthernet0/1 (1), with S-COR-02 GigabitEthernet2/15
>>>> (40). filter f_rewrite_cisco_program_3 {
>>>> match('^\d+[ywdh]\d+[ywdh]: (%[^:]+): ([^\n]+)'
>>>> value("MSGONLY") type("pcre") flags("store-matches"
>>>> "nobackref")); };
>>>> 
>>>> #Feb  6 16:43:32.219: %LINK-3-UPDOWN: Interface 
>>>> FastEthernet2/0/42, changed state to up filter 
>>>> f_rewrite_cisco_program_2 { 
>>>> match('^[\*\.]?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}(?:\.\d+)?(?:
>>>>
>>>>
>
>>>> 
[A-Z]{3})?: (%[^:]+): ([^\n]+)'
>>>> value("MSGONLY") type("pcre") flags("store-matches"
>>>> "nobackref")); };
>>>> 
>>>> # Others where MSGONLY starts with PROGRAM filter 
>>>> f_rewrite_cisco_program { match('^(%[A-Z]+\-\d\-[0-9A-Z]+): 
>>>> ([^\n]+)' value("MSGONLY") type("pcre")
>>>> flags("store-matches" "nobackref")); };
>>>> 
>>>> rewrite r_cisco_program { set("$1", value("PROGRAM") 
>>>> condition(filter(f_rewrite_cisco_program) or 
>>>> filter(f_rewrite_cisco_program_2) or 
>>>> filter(f_rewrite_cisco_program_3))); set("$2",
>>>> value("MESSAGE") condition(filter(f_rewrite_cisco_program)
>>>> or filter(f_rewrite_cisco_program_2) or 
>>>> filter(f_rewrite_cisco_program_3))); };
>>>> 
>>>> This works, but is there a better way? 
>>>> ______________________________________________________________________________
>>>>
>>>>
>
>>>> 
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: 
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>
>>>> 
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>> 
>> ______________________________________________________________________________
>>
>> 
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng 
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> 
> ______________________________________________________________________________
>
> 
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng 
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFPMZ88TCCRT+dccOYRAkD4AJ9RZS5LT44x9oYsNvVEvQpQNgZfQwCePCdP
TVucxhRlGcgD14mRqYMhzD8=
=v6vK
-----END PGP SIGNATURE-----