[syslog-ng] parsing cisco firepower logs problem with 3.33

Mon Feb 28 18:23:21 UTC 2022

Hi Gabor,
Thanks for the reply.  Just looking over the Cisco CSL made my head ache, so I'm glad to hear you think it's complex too.

We're running Cisco NGFW 1140 as the devices sending the data.  Probably running Cisco Firepower 6.6.x of some sort, but I don't have the exact version number handy.  I'll see if I can find it.

If I know what I want to search for, is there a way to quickly write a simple SCL to handle just this format?  I really need to be able to classify by the levels, so the FTD-(\d)-(\d+) where the $1 turns into the level number I can filter against to decide which messages to forward on to another destination would be awesome.

Thanks,
John

Sr. Storage Architect
TOSHIBA AMERICA, INC.
1251 6th,  Ave 41st flr, New York, NY 10020
508-736-5499 (mobile)
E-Mail:  john.stoffel at toshiba.com<mailto:john.stoffel at toshiba.com>
Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do>

From: Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
Sent: Monday, February 28, 2022 5:26 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>; Stoffel, John (TAI) <john.stoffel at toshiba.com>
Subject: Re: parsing cisco firepower logs problem with 3.33

Dear John!

Sorry for not answering earlier.

Thanks for the detailed report of this issue.

To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it.
Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse.
The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination).
Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message.
It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt".

I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp.
It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53).
When I've changed only the timestamp format on one of your log messages, cisco-parser() worked:
<166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01

Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser.

I'll create a pull request about this and discuss it with the team.
Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation.

Regards,
Gabor
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel at toshiba.com<mailto:John.Stoffel at toshiba.com>>
Sent: Thursday, February 17, 2022 15:47
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu> <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi,

I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system.  After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things.

My logs look like this:

<166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr

om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00

<166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr

om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01

<166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from

FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632

Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the:

   sequence, date: origin, %MSG

instead of

  sequence, origin, date: %MSG

and it's not clear to me how I would hack the plugin.conf file to handle this issue.  My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system.

Thanks,

John

Sr. Storage Architect

TOSHIBA AMERICA, INC.

1251 6th,  Ave 41st flr, New York, NY 10020

508-736-5499 (mobile)

E-Mail:  john.stoffel at toshiba.com<mailto:john.stoffel at toshiba.com>

Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=u0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi$>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220228/b4354f40/attachment.htm>