Thanks, a couple of questions. Do you have any blog posts that demonstrate best practices when using junctions and channels? I think I mostly understand it but sometimes things happen that I don't anticipate and I'm trying to figure out what I'm not understanding. Is it possible to give a name to every filter, rewrite, parser, etc.? As I'm looking through traces they are hard to identify when they are anon-filter and such, I have to go look at the line number or figure it out from the context of the log message. I tried this but the --syntax-only check was not having it, perhaps I was doing it wrong. I would also like to be able to write to a file only the log categorization output without the actual log messages, is that possible? It would help me in figuring out why things are not going where they should. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell) From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Peter Czanik (pczanik) Sent: Friday, May 21, 2021 09:09 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: [EXTERNAL] Re: [syslog-ng] Using custom parser with only a subset of udp traffic Hi Syslog-ng, My blog about analyzing Suricata log messages has many examples about processing logs differently based on message content: https://www.syslog-ng.com/community/b/blog/posts/analyze-your-suricata-logs-in-real-time-using-syslog-ng<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Fcommunity%2Fb%2Fblog%2Fposts%2Fanalyze-your-suricata-logs-in-real-time-using-syslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371589019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QkgMtqxNjTOnc4MYMKbL4tRXSf8m5n3FzhLwEAjH8eQ%3D&reserved=0> Have a nice weekend! Peter Peter Czanik (CzP) <peter.czanik@oneidentity.com<mailto:peter.czanik@oneidentity.com>> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsyslog-ng.com%2Fcommunity%2F&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371589019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ABC%2BJ5NEQgeIFNQBnAJ8Gy6bvn2%2B%2FtCxjXTAsE72ZjA%3D&reserved=0> https://twitter.com/PCzanik<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FPCzanik&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371593997%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5naV7dHut3bHkNULsrbNTr0vZ7NAXAQITvwsdUsXoyQ%3D&reserved=0> ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine@nasa.gov<mailto:mark.faine@nasa.gov>> Sent: Friday, May 21, 2021 16:05 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] Using custom parser with only a subset of udp traffic CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. I think I have figured out that I can use one source and always specify the parser manually, that has gotten me most of the way there. Thank you. Still any suggestions for how to proceed are of course welcome. Thanks, -Mark -----Original Message----- From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> On Behalf Of Faine, Mark R. (MSFC-IS40)[NICS] Sent: Thursday, May 20, 2021 09:55 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Using custom parser with only a subset of udp traffic Hopefully one of you can help me figure this out. All of my udp traffic is coming in through one source (network driver), some of it is firewall data but I have to filter it out based on where it is coming from and other fields in the log data. I want to use the csv_parser here since it would make it easier for me to get the data I need from the firewall data. Does this require splitting my one source into two sources, since I would need to use flags('no-parse') on a firewall data source? I also think this means that I can't use a single log path to process both sources? Also, how to prevent potential duplication since the non-firewall source will still receive the firewall traffic and try to process it with syslog-parser. I think the flow is something like this: firewall source -> custom parser -> filters ->rewrites / \ all_data -> \ -> destination \ / all udp source -> filters -> rewrites / I understand this is a bit complex to answer simply, I'm mostly looking for answers to the above questions as well as high level guidance for how to proceed. Thanks, -Mark ______________________________________________________________________________ Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476926994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pfdd17XkUlj53ZWtwjloGxs%2BFCVyoSyCWg%2FsMzni%2FJE%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371598979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=XsqUTuIN8mcwE0NsZTnQ1kPLcMFdRUt6m34UKqSe3XE%3D&reserved=0> Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476926994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZvjFq4y29jjLnicSXcVFbDkstsh7fp2k46juF2fSXNo%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371603954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ITSsd0LE%2BJs12d8nqIjKBgaTwBCxGUfW1dIUNSArF7s%3D&reserved=0> FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=msZEifJqOREtlckmKcBMN4BAlqBeNq3VzRw%2FdsUH%2BM8%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371608932%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mt%2FzxcINhRfSOan9FnLeEuOCRSaA4iDzVZJS23NgDSA%3D&reserved=0> ______________________________________________________________________________ Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qavJ9f%2B988KYI3EMI3oPi7z%2BWJ6e01Kiphqo8aiLTzg%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371613911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IuaTRgk%2FJM6QVXNGHAQyVgXZDFc6G0JS2RQE%2BNeaYkQ%3D&reserved=0> Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vNxB%2B2ayi4V8aOF0SG6VCv4QK2TPrZIiwznujhvhOJw%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371613911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wAAPSFdICCMM6ryNUEtCpiGlQSEnhqmrAv12KZ6se4c%3D&reserved=0> FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=msZEifJqOREtlckmKcBMN4BAlqBeNq3VzRw%2FdsUH%2BM8%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371618889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tF88RPhlM7FcLq35HeMiYBGRq0HvtCkoEWy5ceUMZAs%3D&reserved=0>