Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it's not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do>
Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it’s not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnassc.service-now.com%2Fess%2Fnavpage.do&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7Ce1fc0e410cf542f2294e08d9f22481a5%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637807060893690199%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=u0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0%3D&reserved=0>
Hi Gabor, Thanks for the reply. Just looking over the Cisco CSL made my head ache, so I'm glad to hear you think it's complex too. We're running Cisco NGFW 1140 as the devices sending the data. Probably running Cisco Firepower 6.6.x of some sort, but I don't have the exact version number handy. I'll see if I can find it. If I know what I want to search for, is there a way to quickly write a simple SCL to handle just this format? I really need to be able to classify by the levels, so the FTD-(\d)-(\d+) where the $1 turns into the level number I can filter against to decide which messages to forward on to another destination would be awesome. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Sent: Monday, February 28, 2022 5:26 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>; Stoffel, John (TAI) <john.stoffel@toshiba.com> Subject: Re: parsing cisco firepower logs problem with 3.33 Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it's not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=u0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi$>
Gabor, we're running version 6.7.0 of the Cisco FirePower OS, whatever it's really called. Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Sent: Monday, February 28, 2022 5:26 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>; Stoffel, John (TAI) <john.stoffel@toshiba.com> Subject: Re: parsing cisco firepower logs problem with 3.33 Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it's not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=u0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi$>
Here's a thought... could I just take the existing log files and watch them with a targetted grep command to only get the data I want, and then push that into a new seperate syslog-ng instance to send the data to another remote syslog server? Something like: remote cisco fw -> syslog-ng -> file; * tail -f file| grep "%FTD-1-" | syslog-ng -c /path/to/forwading.conf and have this send only the subset of data I want to forward? I really just need to parse out log files with (in regexp terms) "\s+%FTD-[12]-\d+ \s+" matching the payload, and then just send it on. Any pointers to docs on how I could do this type of stupid silly hack? John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do> From: Stoffel, John (TAI) Sent: Tuesday, March 1, 2022 2:01 PM To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: RE: parsing cisco firepower logs problem with 3.33 Gabor, we're running version 6.7.0 of the Cisco FirePower OS, whatever it's really called. Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>> Sent: Monday, February 28, 2022 5:26 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>; Stoffel, John (TAI) <john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com>> Subject: Re: parsing cisco firepower logs problem with 3.33 Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it's not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=u0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi$>
Sorry for not replying sooner. I'm working on a modified cisco-parser() that acceps ISO timestamps too. I've opened a draft pull request for discussion, but some issues are not yet resolved. https://github.com/syslog-ng/syslog-ng/pull/3934 You mentioned you only need to classify by level/severity (e.g. "%FTD-6-305012"), which means the only essential part for you is the triplet parsing part of the cisco-parser(). You can modify your cisco-parser() implementation to do only that and you can skip the timestamp parsing issue. It won't parse the timestamp from the message, thus your log message will have the received time as timestamp. I've attached an example config, in that you can see a "p_cisco_triplet" parser which has lines copied from the cisco-parser. With that you can classify your log messages based on severity/level. We can improve this workaround if the message format is fix and we don't have to be flexible. I haven't found much in the Cisco documentation, I'm not really a Cisco expert. I was wondering, but is this format the cisco EMBLEM format? [1] I haven't really found any documentation about the format itself. Sorry if this is a bit off-topic. [1] https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/200479-Co... Regards, Gabor ________________________________ From: Stoffel, John (TAI) <John.Stoffel@toshiba.com> Sent: Wednesday, March 2, 2022 20:09 To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: RE: parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Here’s a thought... could I just take the existing log files and watch them with a targetted grep command to only get the data I want, and then push that into a new seperate syslog-ng instance to send the data to another remote syslog server? Something like: remote cisco fw -> syslog-ng -> file; * tail -f file| grep “%FTD-1-“ | syslog-ng -c /path/to/forwading.conf and have this send only the subset of data I want to forward? I really just need to parse out log files with (in regexp terms) “\s+%FTD-[12]-\d+ \s+” matching the payload, and then just send it on. Any pointers to docs on how I could do this type of stupid silly hack? John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnassc.service-now.com%2Fess%2Fnavpage.do&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7C476b081198e54ab9d98608d9fc803442%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637818449832891881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ykLHR6S0KLBVZGwGwQes72bSh%2BRomijg7N9Ev3XkGPo%3D&reserved=0> From: Stoffel, John (TAI) Sent: Tuesday, March 1, 2022 2:01 PM To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: RE: parsing cisco firepower logs problem with 3.33 Gabor, we’re running version 6.7.0 of the Cisco FirePower OS, whatever it’s really called. Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnassc.service-now.com%2Fess%2Fnavpage.do&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7C476b081198e54ab9d98608d9fc803442%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637818449832891881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ykLHR6S0KLBVZGwGwQes72bSh%2BRomijg7N9Ev3XkGPo%3D&reserved=0> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>> Sent: Monday, February 28, 2022 5:26 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>; Stoffel, John (TAI) <john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com>> Subject: Re: parsing cisco firepower logs problem with 3.33 Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it’s not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fnam12.safelinks.protection.outlook.com%2F%3Furl%3Dhttps*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do%26data%3D04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000%26sdata%3Du0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi%24&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7C476b081198e54ab9d98608d9fc803442%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637818449832891881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=p2k%2F7TnO1bggDcKaDZMSFAgkU%2B%2BZwGzJAS15e9jufTM%3D&reserved=0>
Hi Gabor, Do you think we should turn OFF the EMBLEM format, if it's set on our routers? I can ask the network team to do so and we can see what happens... John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Sent: Thursday, March 3, 2022 7:14 AM To: Stoffel, John (TAI) <John.Stoffel@toshiba.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: parsing cisco firepower logs problem with 3.33 Sorry for not replying sooner. I'm working on a modified cisco-parser() that acceps ISO timestamps too. I've opened a draft pull request for discussion, but some issues are not yet resolved. https://github.com/syslog-ng/syslog-ng/pull/3934<https://urldefense.com/v3/__https:/github.com/syslog-ng/syslog-ng/pull/3934__;!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zv-CwdPq$> You mentioned you only need to classify by level/severity (e.g. "%FTD-6-305012"), which means the only essential part for you is the triplet parsing part of the cisco-parser(). You can modify your cisco-parser() implementation to do only that and you can skip the timestamp parsing issue. It won't parse the timestamp from the message, thus your log message will have the received time as timestamp. I've attached an example config, in that you can see a "p_cisco_triplet" parser which has lines copied from the cisco-parser. With that you can classify your log messages based on severity/level. We can improve this workaround if the message format is fix and we don't have to be flexible. I haven't found much in the Cisco documentation, I'm not really a Cisco expert. I was wondering, but is this format the cisco EMBLEM format? [1] I haven't really found any documentation about the format itself. Sorry if this is a bit off-topic. [1] https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/200479-Configure-Logging-on-FTD-via-FMC.html<https://urldefense.com/v3/__https:/www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/200479-Configure-Logging-on-FTD-via-FMC.html__;!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zoB3laud$> Regards, Gabor ________________________________ From: Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Wednesday, March 2, 2022 20:09 To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: RE: parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Here's a thought... could I just take the existing log files and watch them with a targetted grep command to only get the data I want, and then push that into a new seperate syslog-ng instance to send the data to another remote syslog server? Something like: remote cisco fw -> syslog-ng -> file; * tail -f file| grep "%FTD-1-" | syslog-ng -c /path/to/forwading.conf and have this send only the subset of data I want to forward? I really just need to parse out log files with (in regexp terms) "\s+%FTD-[12]-\d+ \s+" matching the payload, and then just send it on. Any pointers to docs on how I could do this type of stupid silly hack? John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=ykLHR6S0KLBVZGwGwQes72bSh*2BRomijg7N9Ev3XkGPo*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zuf9nvCu$> From: Stoffel, John (TAI) Sent: Tuesday, March 1, 2022 2:01 PM To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: RE: parsing cisco firepower logs problem with 3.33 Gabor, we're running version 6.7.0 of the Cisco FirePower OS, whatever it's really called. Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=ykLHR6S0KLBVZGwGwQes72bSh*2BRomijg7N9Ev3XkGPo*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zuf9nvCu$> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>> Sent: Monday, February 28, 2022 5:26 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>; Stoffel, John (TAI) <john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com>> Subject: Re: parsing cisco firepower logs problem with 3.33 Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it's not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2Fnam12.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do*26data*3D04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*26sdata*3Du0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D*26reserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi*24&data=04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=p2k*2F7TnO1bggDcKaDZMSFAgkU*2B*2BZwGzJAS15e9jufTM*3D&reserved=0__;JSUlJSUlJSUlJSoqKioqJSUqKioqKioqKioqKiolJSolJSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zpF1_F0l$>
Hi John! I see you've joined the discussion under https://github.com/syslog-ng/syslog-ng/pull/3934. That's great and welcome! About your question turning off EMBLEM format: I didn't answer your question as I had to do some looking around. I wouldn't suggest turning off EMBLEM format (I don't even know if it's configured or not by the example log). I didn't find a specification about the format, only hints that it requires UDP protocol and it can add the PRI field to the message. Some details are here [1], but otherwise only hints on public forums. What we clearly see is that ISO timestamps in cisco devices are documented, so we should support them (it's documented in [1] too). I would rather suggest switching back the timestamp format on those firewalls, than switching off the emblem format. Or, what about my workaround idea with the extracted cisco-triplet-parser(), was it working? Regards, Gabor [1] https://www.cisco.com/c/en/us/td/docs/security/firepower/660/configuration/g... ________________________________ From: Stoffel, John (TAI) <John.Stoffel@toshiba.com> Sent: Monday, March 7, 2022 19:10 To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: RE: parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi Gabor, Do you think we should turn OFF the EMBLEM format, if it’s set on our routers? I can ask the network team to do so and we can see what happens... John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnassc.service-now.com%2Fess%2Fnavpage.do&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7Cde40164ecbbd4fde059308da0065d31b%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637822734583723132%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bZ1sFvvRUQWuySo6Qi8OPPTyCCbaJ3eyJ3a6%2B3V5QZ4%3D&reserved=0> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Sent: Thursday, March 3, 2022 7:14 AM To: Stoffel, John (TAI) <John.Stoffel@toshiba.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: parsing cisco firepower logs problem with 3.33 Sorry for not replying sooner. I'm working on a modified cisco-parser() that acceps ISO timestamps too. I've opened a draft pull request for discussion, but some issues are not yet resolved. https://github.com/syslog-ng/syslog-ng/pull/3934<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3934__%3B!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zv-CwdPq%24&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7Cde40164ecbbd4fde059308da0065d31b%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637822734583723132%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9lIW1OXbnycdR9oeaQB0LKtMNV2uL9chkHMzt5czNVQ%3D&reserved=0> You mentioned you only need to classify by level/severity (e.g. "%FTD-6-305012"), which means the only essential part for you is the triplet parsing part of the cisco-parser(). You can modify your cisco-parser() implementation to do only that and you can skip the timestamp parsing issue. It won't parse the timestamp from the message, thus your log message will have the received time as timestamp. I've attached an example config, in that you can see a "p_cisco_triplet" parser which has lines copied from the cisco-parser. With that you can classify your log messages based on severity/level. We can improve this workaround if the message format is fix and we don't have to be flexible. I haven't found much in the Cisco documentation, I'm not really a Cisco expert. I was wondering, but is this format the cisco EMBLEM format? [1] I haven't really found any documentation about the format itself. Sorry if this is a bit off-topic. [1] https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/200479-Configure-Logging-on-FTD-via-FMC.html<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fwww.cisco.com%2Fc%2Fen%2Fus%2Fsupport%2Fdocs%2Fsecurity%2Ffirepower-ngfw%2F200479-Configure-Logging-on-FTD-via-FMC.html__%3B!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zoB3laud%24&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7Cde40164ecbbd4fde059308da0065d31b%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637822734583723132%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=zTXa%2Bki5ZMsjJ2Mx%2B0fluGI3DRPrqio3xdpUM227mfc%3D&reserved=0> Regards, Gabor ________________________________ From: Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Wednesday, March 2, 2022 20:09 To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: RE: parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Here’s a thought... could I just take the existing log files and watch them with a targetted grep command to only get the data I want, and then push that into a new seperate syslog-ng instance to send the data to another remote syslog server? Something like: remote cisco fw -> syslog-ng -> file; · tail -f file| grep “%FTD-1-“ | syslog-ng -c /path/to/forwading.conf and have this send only the subset of data I want to forward? I really just need to parse out log files with (in regexp terms) “\s+%FTD-[12]-\d+ \s+” matching the payload, and then just send it on. Any pointers to docs on how I could do this type of stupid silly hack? John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fnam12.safelinks.protection.outlook.com%2F%3Furl%3Dhttps*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do%26data%3D04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000%26sdata%3DykLHR6S0KLBVZGwGwQes72bSh*2BRomijg7N9Ev3XkGPo*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zuf9nvCu%24&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7Cde40164ecbbd4fde059308da0065d31b%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637822734583723132%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ApVaUAvJV9QeJ5bchA%2Fq9rOAKvZ7WzUz3Res4PA55hw%3D&reserved=0> From: Stoffel, John (TAI) Sent: Tuesday, March 1, 2022 2:01 PM To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: RE: parsing cisco firepower logs problem with 3.33 Gabor, we’re running version 6.7.0 of the Cisco FirePower OS, whatever it’s really called. Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fnam12.safelinks.protection.outlook.com%2F%3Furl%3Dhttps*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do%26data%3D04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000%26sdata%3DykLHR6S0KLBVZGwGwQes72bSh*2BRomijg7N9Ev3XkGPo*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zuf9nvCu%24&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7Cde40164ecbbd4fde059308da0065d31b%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637822734583723132%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ApVaUAvJV9QeJ5bchA%2Fq9rOAKvZ7WzUz3Res4PA55hw%3D&reserved=0> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>> Sent: Monday, February 28, 2022 5:26 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>; Stoffel, John (TAI) <john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com>> Subject: Re: parsing cisco firepower logs problem with 3.33 Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it’s not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fnam12.safelinks.protection.outlook.com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2Fnam12.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do*26data*3D04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*26sdata*3Du0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D*26reserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi*24%26data%3D04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000%26sdata%3Dp2k*2F7TnO1bggDcKaDZMSFAgkU*2B*2BZwGzJAS15e9jufTM*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSoqKioqJSUqKioqKioqKioqKiolJSolJSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zpF1_F0l%24&data=04%7C01%7CGabor.Nagy%40oneidentity.com%7Cde40164ecbbd4fde059308da0065d31b%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637822734583723132%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wbbpqBf2YXqIpb50p2%2Fb5IMa1tvHceaHrkZLKkXfFQM%3D&reserved=0>
participants (2)
-
Gabor Nagy (gnagy)
-
Stoffel, John (TAI)