-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Martin, mine is a simplified one of yours (and yes stolen / inspired from elsa;-)) filter f_rewrite_cisco_program { match('%([^:]+):\s+([^\n]+)' value("MESSAGE") type("pcre") flags("store-matches" "nobackref")); }; rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program))); }; just one step and in the rewrite everything before the % gets whiped out. I`m pre-filtering for cisco based logs (which have the mnemonic in the message) and then rewrite only those. I`have logs from different network vendors and some servers too. The combined pre-filter together with the rewrite I newer catched a "false" one. Currently I have around 6-8 k msg/sec and I do not notice an impact on cpu utilization compared to the setup without the rewriting. best regards, Tom On 07.02.2012 22:45, Martin Holste wrote:
Yep, my regexes should account for all of those weird formats. So, your method is a two-step method in which you first clear out everything before the percent sign, then you match from the percent to the colon? Are you only receiving Cisco syslog? Do you have a config example?
On Tue, Feb 7, 2012 at 2:16 PM, Thomas Wollner <tw@wollner-net.de> wrote:
Martin,
the configs on cisco routers are very different. One can use too many options how timestamps are built and that crap makes it to the message part of the log.
Just an example: #service timestamps log myrouter(config)#service timestamps log ? datetime Timestamp with date and time uptime Timestamp with system uptime <cr>
myrouter(config)#service timestamps log date myrouter(config)#service timestamps log datetime ? localtime Use local time zone for timestamps msec Include milliseconds in timestamp show-timezone Add time zone information to timestamp year Include year in timestamp <cr>
Because of the too many options I used to rewrite all that crap with nothing until I find a %FACILITY-PRIORITY-PROGRAM - a so called mnemonic - and set that as the $PROGRAM without leading %. That way I can use a simple regex "everything between % and :" is program and everything until end of line is message.
The above options are just for ios based devices. ios-xe, ios-xr, nx-os are out in the wild now and I`m quite shure they will operate different...
just my 2 cents,
best regards,
Tom
On 07.02.2012 16:26, Martin Holste wrote:
I spent some time yesterday working out the proper regexes to handle the many ways Cisco sends its timestamps, depending on how the device is configured. However, I feel like my solution can be improved upon, so I'd like to see if there's a better way. Here's what I've got so far that seems to be working, though there seems to be a 10-15% CPU penalty at the moment:
#4w4d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/15, changed state to down #1y46w: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet0/1 (1), with S-COR-02 GigabitEthernet2/15 (40). filter f_rewrite_cisco_program_3 { match('^\d+[ywdh]\d+[ywdh]: (%[^:]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); };
#Feb 6 16:43:32.219: %LINK-3-UPDOWN: Interface FastEthernet2/0/42, changed state to up filter f_rewrite_cisco_program_2 { match('^[\*\.]?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}(?:\.\d+)?(?:
[A-Z]{3})?: (%[^:]+): ([^\n]+)'
value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); };
# Others where MSGONLY starts with PROGRAM filter f_rewrite_cisco_program { match('^(%[A-Z]+\-\d\-[0-9A-Z]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); };
rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program) or filter(f_rewrite_cisco_program_2) or filter(f_rewrite_cisco_program_3))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program) or filter(f_rewrite_cisco_program_2) or filter(f_rewrite_cisco_program_3))); };
This works, but is there a better way? ______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFPMZ88TCCRT+dccOYRAkD4AJ9RZS5LT44x9oYsNvVEvQpQNgZfQwCePCdP TVucxhRlGcgD14mRqYMhzD8= =v6vK -----END PGP SIGNATURE-----