PatternDB with CiscoSNMP Module
Thanks for reading, I'm in the process of creating a log to trap for Cisco devices. I have it working for facilities like this: TRINITY-2-TRINITY_SYSLOG_CRIT with this pattern: @ESTRING::%@@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@ Now there are other facilities with an additional hyphen before the severity: EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' which parses into: Testing message: program='EARL-DFC1-1-EXCESSIVE_PARITY_ERROR' message='Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' Match name='.classifier.rule_id', value='09944c71-95eb-4bc0-8575-936931d85715', expected='09944c71-95eb-4bc0-8575-936931d85715' Wrong match name='.cisco.Facility', value='EARL', expected='EARL-DFC1' Wrong match name='.cisco.Severity', value='DFC1', expected='1' Wrong match name='.cisco.MsgName', value='1-EXCESSIVE_PARITY_ERROR', expected='EXCESSIVE_PARITY_ERROR' Match name='.cisco.MsgText', value='EARL 0: Parity error detected in VRAM', expected='EARL 0: Parity error detected in VRAM' I've tried a pattern that would parse EARL and DFC1 into separate variables, then append them into .cisco.Facility - which didn't work. Still that leaves the pattern hardcoded to just two hypens. I've also tried hard coding EARL-DFC1 into the pattern, then use value to set .cisco.Facility correctly. This approach works but being hardcoded, another Facility with embedded hyphens will fail. Ideally I'd like to have a pattern that matches on the last hyphen before the severity and capture that. I could parse using a regexp and setting $1, $2, $3, and $4 to the appropriate variables but I'm concerned about performance issues - I'm thinking this is a last resort solution. Any thoughts on how to proceed? Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover@cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================
Hi, I am in the process of writing a cisco-parser() myself, which uses a combination of csv-parser() and regexps, pattern-db is not very well suited for breaking apart Cisco like conditional structure (which depends on various settings on the cisco side). While handling various settings (service sequence-numbers, service timestamps, logging origin-id, etc), it achieves about 37k msg/sec. What is your target number with regards to performance? Here's my stuff: https://github.com/balabit/syslog-ng/pull/1456 -- Bazsi On Tue, May 2, 2017 at 2:17 AM, Schoonover, Mark E HHHH < Mark.Schoonover@cigna.com> wrote:
Thanks for reading,
I’m in the process of creating a log to trap for Cisco devices. I have it working for facilities like this: TRINITY-2-TRINITY_SYSLOG_CRIT with this pattern:
@ESTRING::%@@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@ @ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@
Now there are other facilities with an additional hyphen before the severity: EARL-DFC1-1-EXCESSIVE_PARITY_ERROR:
Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM'
which parses into:
Testing message: program='EARL-DFC1-1-EXCESSIVE_PARITY_ERROR' message='Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM'
Match name='.classifier.rule_id', value='09944c71-95eb-4bc0-8575-936931d85715', expected='09944c71-95eb-4bc0-8575-936931d85715'
Wrong match name='.cisco.Facility', value='EARL', expected='EARL-DFC1'
Wrong match name='.cisco.Severity', value='DFC1', expected='1'
Wrong match name='.cisco.MsgName', value='1-EXCESSIVE_PARITY_ERROR', expected='EXCESSIVE_PARITY_ERROR'
Match name='.cisco.MsgText', value='EARL 0: Parity error detected in VRAM', expected='EARL 0: Parity error detected in VRAM'
I’ve tried a pattern that would parse EARL and DFC1 into separate variables, then append them into .cisco.Facility – which didn’t work. Still that leaves the pattern hardcoded to just two hypens. I’ve also tried hard coding EARL-DFC1 into the pattern, then use value to set .cisco.Facility correctly. This approach works but being hardcoded, another Facility with embedded hyphens will fail. Ideally I’d like to have a pattern that matches on the last hyphen before the severity and capture that. I could parse using a regexp and setting $1, $2, $3, and $4 to the appropriate variables but I’m concerned about performance issues – I’m thinking this is a last resort solution.
Any thoughts on how to proceed?
Regards,
Mark Schoonover – KA6WKE
Infrastructure Engineering Manager, Splunk Architect
ENE : Tools, Instrumentation and Common Services Team
Office: 32.8697° N, 116.9711° W
Phone : 770-261-7934
Email : mark.schoonover@cigna.com
*HPSM Team: ENE NMS Engineering*
*Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2017 CIGNA*
------------------------------------------------------------ ------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ============================================================ ==================
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks Bazsi, It is a challenge that’s for sure. Looking at the code, I think it’ll parse a Cisco syslog message like this: Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM This snippet csv-parser(delimiters(chars('-')) template("$3") columns('.cisco.facility', '.cisco.severity', '.cisco.mnemonic')); into: .cisco.facility = EARL .cisco.severity = DFC1 .cisco.mnemonic = 1 I’m not familiar with channels and junctions and didn’t find anything in the OSE admin manual. Possibly I’m not fully understanding either. Regards, Mark Schoonover – KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover@cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scheidler, Balázs Sent: Monday, May 01, 2017 11:57 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] PatternDB with CiscoSNMP Module Hi, I am in the process of writing a cisco-parser() myself, which uses a combination of csv-parser() and regexps, pattern-db is not very well suited for breaking apart Cisco like conditional structure (which depends on various settings on the cisco side). While handling various settings (service sequence-numbers, service timestamps, logging origin-id, etc), it achieves about 37k msg/sec. What is your target number with regards to performance? Here's my stuff: https://github.com/balabit/syslog-ng/pull/1456<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_balabit_syslog-2Dng_pull_1456&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=tBE35b_4ndD7X1zSxFEcMXgnwGM1GZ9VB3oIUGF-vtU&e=> -- Bazsi On Tue, May 2, 2017 at 2:17 AM, Schoonover, Mark E HHHH <Mark.Schoonover@cigna.com<mailto:Mark.Schoonover@cigna.com>> wrote: Thanks for reading, I’m in the process of creating a log to trap for Cisco devices. I have it working for facilities like this: TRINITY-2-TRINITY_SYSLOG_CRIT with this pattern: @ESTRING::%@@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@ Now there are other facilities with an additional hyphen before the severity: EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' which parses into: Testing message: program='EARL-DFC1-1-EXCESSIVE_PARITY_ERROR' message='Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' Match name='.classifier.rule_id', value='09944c71-95eb-4bc0-8575-936931d85715', expected='09944c71-95eb-4bc0-8575-936931d85715' Wrong match name='.cisco.Facility', value='EARL', expected='EARL-DFC1' Wrong match name='.cisco.Severity', value='DFC1', expected='1' Wrong match name='.cisco.MsgName', value='1-EXCESSIVE_PARITY_ERROR', expected='EXCESSIVE_PARITY_ERROR' Match name='.cisco.MsgText', value='EARL 0: Parity error detected in VRAM', expected='EARL 0: Parity error detected in VRAM' I’ve tried a pattern that would parse EARL and DFC1 into separate variables, then append them into .cisco.Facility – which didn’t work. Still that leaves the pattern hardcoded to just two hypens. I’ve also tried hard coding EARL-DFC1 into the pattern, then use value to set .cisco.Facility correctly. This approach works but being hardcoded, another Facility with embedded hyphens will fail. Ideally I’d like to have a pattern that matches on the last hyphen before the severity and capture that. I could parse using a regexp and setting $1, $2, $3, and $4 to the appropriate variables but I’m concerned about performance issues – I’m thinking this is a last resort solution. Any thoughts on how to proceed? Regards, Mark Schoonover – KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover@cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ============================================================================== ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=4rnwzztJes-sEVyNI37_YvvyrlHkRLSwjQGprO-fmX0&e=> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=k_tNOdB-fJ6oAIfsJ6wU1JVQv5cTPockc95Diky6exc&e=> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=rNRCDCuEpMjwyl9gta5U2p59odVIqTgghc5xzMfOU_k&e=> ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================
Hi, you can find the description of junctions and channels here: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... and here: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... On Wed, May 3, 2017 at 1:54 AM, Schoonover, Mark E HHHH < Mark.Schoonover@cigna.com> wrote:
Thanks Bazsi,
It is a challenge that’s for sure. Looking at the code, I think it’ll parse a Cisco syslog message like this:
Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM
This snippet
csv-parser(delimiters(chars('-')) template("$3")
columns('.cisco.facility', '.cisco.severity', '.cisco.mnemonic'));
into:
.cisco.facility = EARL
.cisco.severity = DFC1
.cisco.mnemonic = 1
I’m not familiar with channels and junctions and didn’t find anything in the OSE admin manual. Possibly I’m not fully understanding either.
Regards,
Mark Schoonover – KA6WKE
Infrastructure Engineering Manager, Splunk Architect
ENE : Tools, Instrumentation and Common Services Team
Office: 32.8697° N, 116.9711° W
Phone : 770-261-7934 <(770)%20261-7934>
Email : mark.schoonover@cigna.com
*HPSM Team: ENE NMS Engineering*
*Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2017 CIGNA*
*From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Monday, May 01, 2017 11:57 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] PatternDB with CiscoSNMP Module
Hi,
I am in the process of writing a cisco-parser() myself, which uses a combination of csv-parser() and regexps, pattern-db is not very well suited for breaking apart Cisco like conditional structure (which depends on various settings on the cisco side).
While handling various settings (service sequence-numbers, service timestamps, logging origin-id, etc), it achieves about 37k msg/sec. What is your target number with regards to performance?
Here's my stuff: https://github.com/balabit/syslog-ng/pull/1456 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_balabit_syslog-2Dng_pull_1456&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=tBE35b_4ndD7X1zSxFEcMXgnwGM1GZ9VB3oIUGF-vtU&e=>
-- Bazsi
On Tue, May 2, 2017 at 2:17 AM, Schoonover, Mark E HHHH < Mark.Schoonover@cigna.com> wrote:
Thanks for reading,
I’m in the process of creating a log to trap for Cisco devices. I have it working for facilities like this: TRINITY-2-TRINITY_SYSLOG_CRIT with this pattern:
@ESTRING::%@@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@ @ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@
Now there are other facilities with an additional hyphen before the severity: EARL-DFC1-1-EXCESSIVE_PARITY_ERROR:
Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM'
which parses into:
Testing message: program='EARL-DFC1-1-EXCESSIVE_PARITY_ERROR' message='Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM'
Match name='.classifier.rule_id', value='09944c71-95eb-4bc0-8575-936931d85715', expected='09944c71-95eb-4bc0-8575-936931d85715'
Wrong match name='.cisco.Facility', value='EARL', expected='EARL-DFC1'
Wrong match name='.cisco.Severity', value='DFC1', expected='1'
Wrong match name='.cisco.MsgName', value='1-EXCESSIVE_PARITY_ERROR', expected='EXCESSIVE_PARITY_ERROR'
Match name='.cisco.MsgText', value='EARL 0: Parity error detected in VRAM', expected='EARL 0: Parity error detected in VRAM'
I’ve tried a pattern that would parse EARL and DFC1 into separate variables, then append them into .cisco.Facility – which didn’t work. Still that leaves the pattern hardcoded to just two hypens. I’ve also tried hard coding EARL-DFC1 into the pattern, then use value to set .cisco.Facility correctly. This approach works but being hardcoded, another Facility with embedded hyphens will fail. Ideally I’d like to have a pattern that matches on the last hyphen before the severity and capture that. I could parse using a regexp and setting $1, $2, $3, and $4 to the appropriate variables but I’m concerned about performance issues – I’m thinking this is a last resort solution.
Any thoughts on how to proceed?
Regards,
Mark Schoonover – KA6WKE
Infrastructure Engineering Manager, Splunk Architect
ENE : Tools, Instrumentation and Common Services Team
Office: 32.8697° N, 116.9711° W
Phone : 770-261-7934 <(770)%20261-7934>
Email : mark.schoonover@cigna.com
*HPSM Team: ENE NMS Engineering*
*Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2017 CIGNA*
------------------------------------------------------------ ------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ============================================================ ==================
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=4rnwzztJes-sEVyNI37_YvvyrlHkRLSwjQGprO-fmX0&e=> Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=k_tNOdB-fJ6oAIfsJ6wU1JVQv5cTPockc95Diky6exc&e=> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=rNRCDCuEpMjwyl9gta5U2p59odVIqTgghc5xzMfOU_k&e=>
------------------------------------------------------------ ------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ============================================================ ==================
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Just a heads up - Cisco Nexus devices have a different format than normal ios devices, it’s kind of obnoxious. 2017 Apr 26 18:04:43
On May 3, 2017, at 2:26 AM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
Hi, you can find the description of junctions and channels here: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/junctions.html>
and here: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/embedded-objects.html>
On Wed, May 3, 2017 at 1:54 AM, Schoonover, Mark E HHHH <Mark.Schoonover@cigna.com <mailto:Mark.Schoonover@cigna.com>> wrote: Thanks Bazsi,
It is a challenge that’s for sure. Looking at the code, I think it’ll parse a Cisco syslog message like this:
Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM
This snippet
csv-parser(delimiters(chars('-')) template("$3")
columns('.cisco.facility', '.cisco.severity', '.cisco.mnemonic'));
into:
.cisco.facility = EARL
.cisco.severity = DFC1
.cisco.mnemonic = 1
I’m not familiar with channels and junctions and didn’t find anything in the OSE admin manual. Possibly I’m not fully understanding either.
Regards,
Mark Schoonover – KA6WKE
Infrastructure Engineering Manager, Splunk Architect
ENE : Tools, Instrumentation and Common Services Team
Office: 32.8697° N, 116.9711° W
Phone : 770-261-7934 <tel:(770)%20261-7934> Email : mark.schoonover@cigna.com <mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering
Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2017 CIGNA
From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu>] On Behalf Of Scheidler, Balázs Sent: Monday, May 01, 2017 11:57 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] PatternDB with CiscoSNMP Module
Hi,
I am in the process of writing a cisco-parser() myself, which uses a combination of csv-parser() and regexps, pattern-db is not very well suited for breaking apart Cisco like conditional structure (which depends on various settings on the cisco side).
While handling various settings (service sequence-numbers, service timestamps, logging origin-id, etc), it achieves about 37k msg/sec. What is your target number with regards to performance?
Here's my stuff: https://github.com/balabit/syslog-ng/pull/1456 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_balabit_syslog-2Dng_pull_1456&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=tBE35b_4ndD7X1zSxFEcMXgnwGM1GZ9VB3oIUGF-vtU&e=>
-- Bazsi
On Tue, May 2, 2017 at 2:17 AM, Schoonover, Mark E HHHH <Mark.Schoonover@cigna.com <mailto:Mark.Schoonover@cigna.com>> wrote:
Thanks for reading,
I’m in the process of creating a log to trap for Cisco devices. I have it working for facilities like this: TRINITY-2-TRINITY_SYSLOG_CRIT with this pattern:
@ESTRING::%@@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@
Now there are other facilities with an additional hyphen before the severity: EARL-DFC1-1-EXCESSIVE_PARITY_ERROR:
Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM'
which parses into:
Testing message: program='EARL-DFC1-1-EXCESSIVE_PARITY_ERROR' message='Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM'
Match name='.classifier.rule_id', value='09944c71-95eb-4bc0-8575-936931d85715', expected='09944c71-95eb-4bc0-8575-936931d85715'
Wrong match name='.cisco.Facility', value='EARL', expected='EARL-DFC1'
Wrong match name='.cisco.Severity', value='DFC1', expected='1'
Wrong match name='.cisco.MsgName', value='1-EXCESSIVE_PARITY_ERROR', expected='EXCESSIVE_PARITY_ERROR'
Match name='.cisco.MsgText', value='EARL 0: Parity error detected in VRAM', expected='EARL 0: Parity error detected in VRAM'
I’ve tried a pattern that would parse EARL and DFC1 into separate variables, then append them into .cisco.Facility – which didn’t work. Still that leaves the pattern hardcoded to just two hypens. I’ve also tried hard coding EARL-DFC1 into the pattern, then use value to set .cisco.Facility correctly. This approach works but being hardcoded, another Facility with embedded hyphens will fail. Ideally I’d like to have a pattern that matches on the last hyphen before the severity and capture that. I could parse using a regexp and setting $1, $2, $3, and $4 to the appropriate variables but I’m concerned about performance issues – I’m thinking this is a last resort solution.
Any thoughts on how to proceed?
Regards,
Mark Schoonover – KA6WKE
Infrastructure Engineering Manager, Splunk Architect
ENE : Tools, Instrumentation and Common Services Team
Office: 32.8697° N, 116.9711° W
Phone : 770-261-7934 <tel:(770)%20261-7934> Email : mark.schoonover@cigna.com <mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering
Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2017 CIGNA
------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=4rnwzztJes-sEVyNI37_YvvyrlHkRLSwjQGprO-fmX0&e=> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=k_tNOdB-fJ6oAIfsJ6wU1JVQv5cTPockc95Diky6exc&e=> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=rNRCDCuEpMjwyl9gta5U2p59odVIqTgghc5xzMfOU_k&e=>
------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
We have Nexus in our environment too. One observation about cisco-snmp; I noticed in the example pattern, that the device name is not parsed and set to a variable. Does this mean all traps sent contain the hostname of the Syslog-NG server or is the device name set elsewhere? Regards, Mark Schoonover – KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover@cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Nik Ambrosch Sent: Wednesday, May 03, 2017 9:09 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] PatternDB with CiscoSNMP Module Just a heads up - Cisco Nexus devices have a different format than normal ios devices, it’s kind of obnoxious. 2017 Apr 26 18:04:43 On May 3, 2017, at 2:26 AM, Fekete, Róbert <robert.fekete@balabit.com<mailto:robert.fekete@balabit.com>> wrote: Hi, you can find the description of junctions and channels here: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/junctions.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.balabit.com_documents_syslog-2Dng-2Dose-2Dlatest-2Dguides_en_syslog-2Dng-2Dose-2Dguide-2Dadmin_html_junctions.html&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=jLnDb6sQhYzSo3Lz2wSHZ5yOPWaOozOG63bY1qTri-I&e=> and here: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/embedded-objects.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.balabit.com_documents_syslog-2Dng-2Dose-2Dlatest-2Dguides_en_syslog-2Dng-2Dose-2Dguide-2Dadmin_html_embedded-2Dobjects.html&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=5oSTqqi5Cs34YGgy-rcDuyOqvkze4BcUFylsHOuBXAc&e=> On Wed, May 3, 2017 at 1:54 AM, Schoonover, Mark E HHHH <Mark.Schoonover@cigna.com<mailto:Mark.Schoonover@cigna.com>> wrote: Thanks Bazsi, It is a challenge that’s for sure. Looking at the code, I think it’ll parse a Cisco syslog message like this: Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM This snippet csv-parser(delimiters(chars('-')) template("$3") columns('.cisco.facility', '.cisco.severity', '.cisco.mnemonic')); into: .cisco.facility = EARL .cisco.severity = DFC1 .cisco.mnemonic = 1 I’m not familiar with channels and junctions and didn’t find anything in the OSE admin manual. Possibly I’m not fully understanding either. Regards, Mark Schoonover – KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934<tel:(770)%20261-7934> Email : mark.schoonover@cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>] On Behalf Of Scheidler, Balázs Sent: Monday, May 01, 2017 11:57 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] PatternDB with CiscoSNMP Module Hi, I am in the process of writing a cisco-parser() myself, which uses a combination of csv-parser() and regexps, pattern-db is not very well suited for breaking apart Cisco like conditional structure (which depends on various settings on the cisco side). While handling various settings (service sequence-numbers, service timestamps, logging origin-id, etc), it achieves about 37k msg/sec. What is your target number with regards to performance? Here's my stuff: https://github.com/balabit/syslog-ng/pull/1456<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_balabit_syslog-2Dng_pull_1456&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=tBE35b_4ndD7X1zSxFEcMXgnwGM1GZ9VB3oIUGF-vtU&e=> -- Bazsi On Tue, May 2, 2017 at 2:17 AM, Schoonover, Mark E HHHH <Mark.Schoonover@cigna.com<mailto:Mark.Schoonover@cigna.com>> wrote: Thanks for reading, I’m in the process of creating a log to trap for Cisco devices. I have it working for facilities like this: TRINITY-2-TRINITY_SYSLOG_CRIT with this pattern: @ESTRING::%@@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@ Now there are other facilities with an additional hyphen before the severity: EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' which parses into: Testing message: program='EARL-DFC1-1-EXCESSIVE_PARITY_ERROR' message='Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' Match name='.classifier.rule_id', value='09944c71-95eb-4bc0-8575-936931d85715', expected='09944c71-95eb-4bc0-8575-936931d85715' Wrong match name='.cisco.Facility', value='EARL', expected='EARL-DFC1' Wrong match name='.cisco.Severity', value='DFC1', expected='1' Wrong match name='.cisco.MsgName', value='1-EXCESSIVE_PARITY_ERROR', expected='EXCESSIVE_PARITY_ERROR' Match name='.cisco.MsgText', value='EARL 0: Parity error detected in VRAM', expected='EARL 0: Parity error detected in VRAM' I’ve tried a pattern that would parse EARL and DFC1 into separate variables, then append them into .cisco.Facility – which didn’t work. Still that leaves the pattern hardcoded to just two hypens. I’ve also tried hard coding EARL-DFC1 into the pattern, then use value to set .cisco.Facility correctly. This approach works but being hardcoded, another Facility with embedded hyphens will fail. Ideally I’d like to have a pattern that matches on the last hyphen before the severity and capture that. I could parse using a regexp and setting $1, $2, $3, and $4 to the appropriate variables but I’m concerned about performance issues – I’m thinking this is a last resort solution. Any thoughts on how to proceed? Regards, Mark Schoonover – KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934<tel:(770)%20261-7934> Email : mark.schoonover@cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ============================================================================== ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=4rnwzztJes-sEVyNI37_YvvyrlHkRLSwjQGprO-fmX0&e=> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=k_tNOdB-fJ6oAIfsJ6wU1JVQv5cTPockc95Diky6exc&e=> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=s7SsNLIK6w8eENvA6fSVfIY_hmH_Ao37Vfkb8LMT5go&s=rNRCDCuEpMjwyl9gta5U2p59odVIqTgghc5xzMfOU_k&e=> ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ============================================================================== ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=sQuG-iwFD4EJq-XAqPSZ0xRyMuiRok2f4xH1-OZiwcs&e=> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=gTmfLgR4wEMhmm_W_T18mqRlBY7L9nvumAzKLGAkL_w&e=> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=fPuXmLjYqxKUPnktrDdJUNAYxomoifNDzFf4lIfduQY&e=> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=sQuG-iwFD4EJq-XAqPSZ0xRyMuiRok2f4xH1-OZiwcs&e=> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=gTmfLgR4wEMhmm_W_T18mqRlBY7L9nvumAzKLGAkL_w&e=> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwMFaQ&c=WlnXFIBCT3pmNh_w8hYeLw&r=7hmZ2rDjdhHVbv2tovCmuIwqUlXIJdVXqIASPRODb2U&m=h4qg7O-vmHrn8Ck9fHdQ1CUHQ40kM1zA-jHhf_RygvE&s=fPuXmLjYqxKUPnktrDdJUNAYxomoifNDzFf4lIfduQY&e=> ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================
participants (4)
-
Fekete, Róbert
-
Nik Ambrosch
-
Scheidler, Balázs
-
Schoonover, Mark E HHHH