Hi, Digging around, I could not find anything on how to 'sensibly' log the junk that Cisco IOS devices (and their infernal WLC) spit out plus I was keen to rewrite squid HTTP proxy server logs to make use of the epoch+msec timestamp found in the MSG. I have documented, very roughly, my current solution on my website and would welcome amendments/fixes/flames/etc on the approach: http://www.digriz.org.uk/syslog-ng-integration Before you ask, I use 'match("fqdn.example.com" value("HOST_FROM") ...)' as netmask() is broken for IPv6 :) Cheers -- Alexander Clouter .sigmonster says: When pleasure remains, does it remain a pleasure?
Cisco messages are easy to log than most IMHO. The great thing about them is that they include the starting Mnemonic (Facility-Severity-Mnemonic). Take a look at my whitepaper here: http://www.cisco.com/en/US/technologies/collateral/tk869/tk769/white_paper_c... I also use these to track Cisco events in my tool, LogZila ( http://www.logzilla.pro). The great thing about them is that by tracking them, you can instantly get a view of your network problems by generating graphs of top 10 mnemonics. You can see what I mean by looking at the demo site at http://demo/logzilla.pro - once there, just click on "Charts>Top10>By Count>Cisco Mnemonics". ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Mon, Mar 7, 2011 at 6:35 AM, Alexander Clouter <alex@digriz.org.uk>wrote:
Hi,
Digging around, I could not find anything on how to 'sensibly' log the junk that Cisco IOS devices (and their infernal WLC) spit out plus I was keen to rewrite squid HTTP proxy server logs to make use of the epoch+msec timestamp found in the MSG.
I have documented, very roughly, my current solution on my website and would welcome amendments/fixes/flames/etc on the approach:
http://www.digriz.org.uk/syslog-ng-integration
Before you ask, I use 'match("fqdn.example.com" value("HOST_FROM") ...)' as netmask() is broken for IPv6 :)
Cheers
-- Alexander Clouter .sigmonster says: When pleasure remains, does it remain a pleasure?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi, I think you have missed what I have set out to accomplish, digesting and log analysis is not the problem I'm solving[1] * Clayton Dukes <cdukes@gmail.com> [2011-03-07 08:40:55-0500]:
Cisco messages are easy to log than most IMHO.
The sequence number and scattering of '*'/'.' infront of the timestamp makes it anything but easy to log; especially if you want to trust the sending host's timestamp and have all your output logs in a *standard* format[2]. Cisco devices do *not* send messages in a format syslog-ng cannot parse directly (or not one I have found). Why does IOS sometimes put a '.' infront of the date and other times does not? The only helpful bit I got from your whitepaper is now I know what '*' means, no idea why you did not just append '+02:30' or whatever on the date instead? I'm trying to normalise the cruft IOS sends me, not analyse it. Once it is in a standard format I can use generic shell/perl scripts to parse the contents, rather than custom Cisco-only scripts. Cheers [1] I actually prefer a daily cronjob of various types of 'catches of the day', generated from awk/perl scripts that get dumped into my mailbox. For example, 'top ten' egress user IP's appearing in the firewall. This is just how I like to butter by bread though :) [2] I really like the output from "$ISODATE $FULLHOST <$FACILITY.$PRIORITY> $MSGHDR$MSGONLY" -- Alexander Clouter .sigmonster says: Thank God I'm an atheist.
The * and . characters are NTP problems - they mean that your devices are not configured/synched properly: Symbol Description * Time is not authoritative: the software clock is not in sync or has never been set. (blank) Time is authoritative: the software clock is in sync or has just been set manually . Time is authoritative, but NTP is not synchronized: the software clock was in sync, but has since lost contact with all configured NTP servers I'm using: $S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n I use tabs as a delimiter, but of course you can use the delim of your choice :-) In my parser, I use: my $re_pipe = qr/(\S+ \S+)\t(\S+)\t(\d+)\t(\S+).*\t(.*)/; my $re_mne = qr/\%([A-Z\-\d\_]+?\-\d+\-[A-Z\-\_\d]+?)(?:\:|\s)/; # Cisco Mnemonics capture ...while loop: # v3.2 Fields are: TS, Host, PRI, Program, and MSG if ($msg =~ m/$re_pipe/) { $ts = $1; $host = $2; $pri = $3; $facility = int($pri/8); $severity = $pri - ($facility * 8 ); $prg = $4; $msg = $5; HTH :-) ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Mon, Mar 7, 2011 at 9:07 AM, Alexander Clouter <alex@digriz.org.uk>wrote:
Hi,
I think you have missed what I have set out to accomplish, digesting and log analysis is not the problem I'm solving[1]
* Clayton Dukes <cdukes@gmail.com> [2011-03-07 08:40:55-0500]:
Cisco messages are easy to log than most IMHO.
The sequence number and scattering of '*'/'.' infront of the timestamp makes it anything but easy to log; especially if you want to trust the sending host's timestamp and have all your output logs in a *standard* format[2].
Cisco devices do *not* send messages in a format syslog-ng cannot parse directly (or not one I have found). Why does IOS sometimes put a '.' infront of the date and other times does not? The only helpful bit I got from your whitepaper is now I know what '*' means, no idea why you did not just append '+02:30' or whatever on the date instead?
I'm trying to normalise the cruft IOS sends me, not analyse it. Once it is in a standard format I can use generic shell/perl scripts to parse the contents, rather than custom Cisco-only scripts.
Cheers
[1] I actually prefer a daily cronjob of various types of 'catches of the day', generated from awk/perl scripts that get dumped into my mailbox. For example, 'top ten' egress user IP's appearing in the firewall. This is just how I like to butter by bread though :) [2] I really like the output from "$ISODATE $FULLHOST <$FACILITY.$PRIORITY> $MSGHDR$MSGONLY"
-- Alexander Clouter .sigmonster says: Thank God I'm an atheist.
Hi, * Clayton Dukes <cdukes@gmail.com> [2011-03-07 09:28:35-0500]:
The * and . characters are NTP problems - they mean that your devices are not configured/synched properly: Symbol Description
but but but... http://www.cisco.com/en/US/technologies/collateral/tk869/tk769/white_paper_c... "Cisco IOS Software allows devices to be configured to send the time-zone information in the message part of the Syslog packet. When this occurs, the message will be marked with an asterisk (*)." Or does 'message' here mean $MSG?
* Time is not authoritative: the software clock is not in sync or has never been set.
(blank) Time is authoritative: the software clock is in sync or has just been set manually
. Time is authoritative, but NTP is not synchronized: the software clock was in sync, but has since lost contact with all configured NTP servers
/me adds another thing to his todo list of things to fix :-/ In my defence, I cannot find where this documented on the Cisco website.
I'm using: $S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n
I use tabs as a delimiter, but of course you can use the delim of your choice :-)
In my parser, I use: my $re_pipe = qr/(\S+ \S+)\t(\S+)\t(\d+)\t(\S+).*\t(.*)/; my $re_mne = qr/\%([A-Z\-\d\_]+?\-\d+\-[A-Z\-\_\d]+?)(?:\:|\s)/; # Cisco Mnemonics capture
...while loop: # v3.2 Fields are: TS, Host, PRI, Program, and MSG if ($msg =~ m/$re_pipe/) { $ts = $1; $host = $2; $pri = $3; $facility = int($pri/8); $severity = $pri - ($facility * 8 ); $prg = $4; $msg = $5;
Much like the squid rewriter on the page, I did use a perl script originally, then worked out how (with a monkey wrench) I could persuade syslog-ng to do my dirty work :) Cheers -- Alexander Clouter .sigmonster says: Life is cheap, but the accessories can kill you.
but but but...
http://www.cisco.com/en/US/technologies/collateral/tk869/tk769/white_paper_c... <http://www.cisco.com/en/US/technologies/collateral/tk869/tk769/white_paper_c11-557812.html#wp9000339>Yep...I'm trying to get the WP corrected (which was based on another incorrect document at cisco.com), but it's difficult to get WP's updated (large company, trying to find out *who* can make the change).
In my defence, I cannot find where this documented on the Cisco website. Can't say I blame you :-)
______________________________________________________________ Clayton Dukes ______________________________________________________________ On Mon, Mar 7, 2011 at 10:58 AM, Alexander Clouter <alex@digriz.org.uk>wrote:
Hi,
* Clayton Dukes <cdukes@gmail.com> [2011-03-07 09:28:35-0500]:
The * and . characters are NTP problems - they mean that your devices are not configured/synched properly: Symbol Description
but but but...
http://www.cisco.com/en/US/technologies/collateral/tk869/tk769/white_paper_c...
"Cisco IOS Software allows devices to be configured to send the time-zone information in the message part of the Syslog packet. When this occurs, the message will be marked with an asterisk (*)."
Or does 'message' here mean $MSG?
* Time is not authoritative: the software clock is not in sync or has never been set.
(blank) Time is authoritative: the software clock is in sync or has just been set manually
. Time is authoritative, but NTP is not synchronized: the software clock was in sync, but has since lost contact with all configured NTP servers
/me adds another thing to his todo list of things to fix :-/
In my defence, I cannot find where this documented on the Cisco website.
I'm using: $S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n
I use tabs as a delimiter, but of course you can use the delim of your choice :-)
In my parser, I use: my $re_pipe = qr/(\S+ \S+)\t(\S+)\t(\d+)\t(\S+).*\t(.*)/; my $re_mne = qr/\%([A-Z\-\d\_]+?\-\d+\-[A-Z\-\_\d]+?)(?:\:|\s)/; # Cisco Mnemonics capture
...while loop: # v3.2 Fields are: TS, Host, PRI, Program, and MSG if ($msg =~ m/$re_pipe/) { $ts = $1; $host = $2; $pri = $3; $facility = int($pri/8); $severity = $pri - ($facility * 8 ); $prg = $4; $msg = $5;
Much like the squid rewriter on the page, I did use a perl script originally, then worked out how (with a monkey wrench) I could persuade syslog-ng to do my dirty work :)
Cheers
-- Alexander Clouter .sigmonster says: Life is cheap, but the accessories can kill you.
participants (2)
-
Alexander Clouter
-
Clayton Dukes