[syslog-ng] parsing cisco firepower logs problem with 3.33

Mon Mar 7 18:10:52 UTC 2022

Hi Gabor,
Do you think we should turn OFF the EMBLEM format, if it's set on our routers?  I can ask the network team to do so and we can see what happens...

John

Sr. Storage Architect
TOSHIBA AMERICA, INC.
1251 6th,  Ave 41st flr, New York, NY 10020
508-736-5499 (mobile)
E-Mail:  john.stoffel at toshiba.com<mailto:john.stoffel at toshiba.com>
Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do>

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
Sent: Thursday, March 3, 2022 7:14 AM
To: Stoffel, John (TAI) <John.Stoffel at toshiba.com>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: parsing cisco firepower logs problem with 3.33

Sorry for not replying sooner.

I'm working on a modified cisco-parser() that acceps ISO timestamps too.
I've opened a draft pull request for discussion, but some issues are not yet resolved.
https://github.com/syslog-ng/syslog-ng/pull/3934<https://urldefense.com/v3/__https:/github.com/syslog-ng/syslog-ng/pull/3934__;!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zv-CwdPq$>

You mentioned you only need to classify by level/severity (e.g. "%FTD-6-305012"), which means the only essential part for you is the triplet parsing part of the cisco-parser().
You can modify your cisco-parser() implementation to do only that and you can skip the timestamp parsing issue.
It won't parse the timestamp from the message, thus your log message will have the received time as timestamp.
I've attached an example config, in that you can see a "p_cisco_triplet" parser which has lines copied from the cisco-parser.
With that you can classify your log messages based on severity/level.
We can improve this workaround if the message format is fix and we don't have to be flexible.

I haven't found much in the Cisco documentation,  I'm not really a Cisco expert.
I was wondering, but is this format the cisco EMBLEM format? [1]
I haven't really found any documentation about the format itself. Sorry if this is a bit off-topic.

[1] https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/200479-Configure-Logging-on-FTD-via-FMC.html<https://urldefense.com/v3/__https:/www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/200479-Configure-Logging-on-FTD-via-FMC.html__;!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zoB3laud$>

Regards,
Gabor
________________________________
From: Stoffel, John (TAI) <John.Stoffel at toshiba.com<mailto:John.Stoffel at toshiba.com>>
Sent: Wednesday, March 2, 2022 20:09
To: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: RE: parsing cisco firepower logs problem with 3.33

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Here's a thought... could I just take the existing log files and watch them with a targetted grep command to only get the data I want, and then push that into a new seperate syslog-ng instance to send the data to another remote syslog server?

Something like:

      remote cisco fw -> syslog-ng -> file;

*         tail -f file| grep "%FTD-1-" | syslog-ng -c /path/to/forwading.conf

and have this send only the subset of data I want to forward?  I really just need to parse out log files with (in regexp terms) "\s+%FTD-[12]-\d+ \s+" matching the payload, and then just send it on.

Any pointers to docs on how I could do this type of stupid silly hack?

John

Sr. Storage Architect

TOSHIBA AMERICA, INC.

1251 6th,  Ave 41st flr, New York, NY 10020

508-736-5499 (mobile)

E-Mail:  john.stoffel at toshiba.com<mailto:john.stoffel at toshiba.com>

Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=ykLHR6S0KLBVZGwGwQes72bSh*2BRomijg7N9Ev3XkGPo*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zuf9nvCu$>

From: Stoffel, John (TAI)
Sent: Tuesday, March 1, 2022 2:01 PM
To: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: RE: parsing cisco firepower logs problem with 3.33

Gabor, we're running version 6.7.0 of the Cisco FirePower OS, whatever it's really called.

Sr. Storage Architect

TOSHIBA AMERICA, INC.

1251 6th,  Ave 41st flr, New York, NY 10020

508-736-5499 (mobile)

E-Mail:  john.stoffel at toshiba.com<mailto:john.stoffel at toshiba.com>

Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=ykLHR6S0KLBVZGwGwQes72bSh*2BRomijg7N9Ev3XkGPo*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zuf9nvCu$>

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com<mailto:Gabor.Nagy at oneidentity.com>>
Sent: Monday, February 28, 2022 5:26 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>; Stoffel, John (TAI) <john.stoffel at toshiba.com<mailto:john.stoffel at toshiba.com>>
Subject: Re: parsing cisco firepower logs problem with 3.33

Dear John!

Sorry for not answering earlier.

Thanks for the detailed report of this issue.

To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it.
Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse.

The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination).

Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message.

It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt".

I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp.
It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53).

When I've changed only the timestamp format on one of your log messages, cisco-parser() worked:
<166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01

Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser.

I'll create a pull request about this and discuss it with the team.

Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation.

Regards,
Gabor

________________________________

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel at toshiba.com<mailto:John.Stoffel at toshiba.com>>
Sent: Thursday, February 17, 2022 15:47
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu> <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi,

I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system.  After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things.

My logs look like this:

<166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr

om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00

<166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr

om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01

<166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from

FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632

Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the:

   sequence, date: origin, %MSG

instead of

  sequence, origin, date: %MSG

and it's not clear to me how I would hack the plugin.conf file to handle this issue.  My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system.

Thanks,

John

Sr. Storage Architect

TOSHIBA AMERICA, INC.

1251 6th,  Ave 41st flr, New York, NY 10020

508-736-5499 (mobile)

E-Mail:  john.stoffel at toshiba.com<mailto:john.stoffel at toshiba.com>

Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2Fnam12.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do*26data*3D04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*26sdata*3Du0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D*26reserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi*24&data=04*7C01*7CGabor.Nagy*40oneidentity.com*7C476b081198e54ab9d98608d9fc803442*7C91c369b51c9e439c989c1867ec606603*7C0*7C0*7C637818449832891881*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=p2k*2F7TnO1bggDcKaDZMSFAgkU*2B*2BZwGzJAS15e9jufTM*3D&reserved=0__;JSUlJSUlJSUlJSoqKioqJSUqKioqKioqKioqKiolJSolJSUlJSUlJSUlJSUlJSUlJSUlJQ!!BiNunAf9XXY-!TCbZSyBA4z7eEUxKNDYsxG8ay0PE23yHIdo0ZEN_kkIYpxfLEYODdeS-E2w9zpF1_F0l$>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220307/afef3160/attachment-0001.htm>