Here's a thought... could I just take the existing log files and watch them with a targetted grep command to only get the data I want, and then push that into a new seperate syslog-ng instance to send the data to another remote syslog server? Something like: remote cisco fw -> syslog-ng -> file; * tail -f file| grep "%FTD-1-" | syslog-ng -c /path/to/forwading.conf and have this send only the subset of data I want to forward? I really just need to parse out log files with (in regexp terms) "\s+%FTD-[12]-\d+ \s+" matching the payload, and then just send it on. Any pointers to docs on how I could do this type of stupid silly hack? John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do> From: Stoffel, John (TAI) Sent: Tuesday, March 1, 2022 2:01 PM To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: RE: parsing cisco firepower logs problem with 3.33 Gabor, we're running version 6.7.0 of the Cisco FirePower OS, whatever it's really called. Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do> From: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com<mailto:Gabor.Nagy@oneidentity.com>> Sent: Monday, February 28, 2022 5:26 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>; Stoffel, John (TAI) <john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com>> Subject: Re: parsing cisco firepower logs problem with 3.33 Dear John! Sorry for not answering earlier. Thanks for the detailed report of this issue. To be honest, cisco-parser is probably the most complex SCL in syslog-ng, and it's hard to debug it. Message processing can be debugged if syslog-ng is running with trace-level debugging, but it's not an easy output to parse. The internal logs show what happens to a log message on each pipeline element (from sources until it reaches the destination). Trace level internal logs causes vast amount of logs on the console or internal() log, so I recommend using this only for debugging 1 message. It can be turned on via "syslog-ng-ctl trace -s 1" or starting syslog-ng in the foreground: "syslog-ng -Fedvt". I've checked the log formats you sent us, and the main problem is not with the order of elements, but the format of the timestamp. It's an ISO-8601 formatted timestamp, while the cisco-parser only supports the old "day-name month" format (e.g. Feb 16 2022 16:31:53). When I've changed only the timestamp format on one of your log messages, cisco-parser() worked: <166>Feb 16 2022 16:31:53 na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation from FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 Also with the changed order the hostname (or by Cisco terminology "origin-id") cannot be parsed by the cisco-parser. I'll create a pull request about this and discuss it with the team. Can you send us some information about that Cisco device that sends these logs, please? So we can look into it's documentation. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Stoffel, John (TAI) <John.Stoffel@toshiba.com<mailto:John.Stoffel@toshiba.com>> Sent: Thursday, February 17, 2022 15:47 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] parsing cisco firepower logs problem with 3.33 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it's not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel@toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fnassc.service-now.com*2Fess*2Fnavpage.do&data=04*7C01*7Cgabor.nagy*40oneidentity.com*7Ce1fc0e410cf542f2294e08d9f22481a5*7C91c369b51c9e439c989c1867ec606603*7C0*7C1*7C637807060893690199*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=u0eNB5EHzsyTSOvNbI7czRJLxpvC2EPeeKsZ6H5X9q0*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUl!!BiNunAf9XXY-!R4NbMeGvRLi2JniMHFDJNW1kydS0JyHKyMA48a4Y9i-LYsY-BKG3QcjH71lz5Iw8hNbi$>