[syslog-ng] [EXTERNAL] Re: Using custom parser with only a subset of udp traffic

Fri May 21 14:22:09 UTC 2021

Thanks, a couple of questions.  Do you have any blog posts that demonstrate best practices when using junctions and channels?  I think I mostly understand it but sometimes things happen that I don't anticipate and I'm trying to figure out what I'm not understanding.

Is it possible to give a name to every filter, rewrite, parser, etc.?  As I'm looking through traces they are hard to identify when they are anon-filter and such, I have to go look at the line number or figure it out from the context of the log message.  I tried this but the --syntax-only check was not having it, perhaps I was doing it wrong.  I would also like to be able to write to a file only the log categorization output without the actual log messages, is that possible?  It would help me in figuring out why things are not going where they should.

Thanks,
-Mark

Mark Faine
System Administrator
SAIC/NICS
215 Wynn Dr. 5065
Huntsville, AL 35805
256-961-1295 (Desk)
256-617-4861 (Work Cell)

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Peter Czanik (pczanik)
Sent: Friday, May 21, 2021 09:09
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [EXTERNAL] Re: [syslog-ng] Using custom parser with only a subset of udp traffic

Hi Syslog-ng,

My blog about analyzing Suricata log messages has many examples about processing logs differently based on message content: https://www.syslog-ng.com/community/b/blog/posts/analyze-your-suricata-logs-in-real-time-using-syslog-ng<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Fcommunity%2Fb%2Fblog%2Fposts%2Fanalyze-your-suricata-logs-in-real-time-using-syslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371589019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QkgMtqxNjTOnc4MYMKbL4tRXSf8m5n3FzhLwEAjH8eQ%3D&reserved=0>

Have a nice weekend!
Peter

Peter Czanik (CzP) <peter.czanik at oneidentity.com<mailto:peter.czanik at oneidentity.com>>
Balabit (a OneIdentity company) / syslog-ng upstream
https://syslog-ng.com/community/<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsyslog-ng.com%2Fcommunity%2F&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371589019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ABC%2BJ5NEQgeIFNQBnAJ8Gy6bvn2%2B%2FtCxjXTAsE72ZjA%3D&reserved=0>
https://twitter.com/PCzanik<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FPCzanik&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371593997%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5naV7dHut3bHkNULsrbNTr0vZ7NAXAQITvwsdUsXoyQ%3D&reserved=0>

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov<mailto:mark.faine at nasa.gov>>
Sent: Friday, May 21, 2021 16:05
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] Using custom parser with only a subset of udp traffic

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

I think I have figured out that I can use one source and always specify the parser manually, that has gotten me most of the way there.  Thank you. Still any suggestions for how to proceed are of course welcome.

Thanks,
-Mark

-----Original Message-----
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Faine, Mark R. (MSFC-IS40)[NICS]
Sent: Thursday, May 20, 2021 09:55
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] Using custom parser with only a subset of udp traffic

Hopefully one of you can help me figure this out.   All of my udp traffic is coming in through one source (network driver), some of it is firewall data but I have to filter it out based on where it is coming from and other fields in the log data.  I want to use the csv_parser here since it would make it easier for me to get the data I need from the firewall data.  Does this require splitting my one source into two sources, since I would need to use flags('no-parse') on a firewall data source?   I also think this means that I can't use a single log path to process both sources?  Also, how to prevent potential duplication since the non-firewall source will still receive the firewall traffic and try to process it with syslog-parser.

I think the flow is something like this:

                        firewall source -> custom parser -> filters ->rewrites
                      /                                                                                          \
all_data ->                                                                                                 \  -> destination
                    \                                                                                            /
                     all udp source -> filters -> rewrites                             /

I understand this is a bit complex to answer simply, I'm mostly looking for answers to the above questions as well as high level guidance for how to proceed.

Thanks,
-Mark
______________________________________________________________________________
Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476926994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pfdd17XkUlj53ZWtwjloGxs%2BFCVyoSyCWg%2FsMzni%2FJE%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371598979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=XsqUTuIN8mcwE0NsZTnQ1kPLcMFdRUt6m34UKqSe3XE%3D&reserved=0>
Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476926994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZvjFq4y29jjLnicSXcVFbDkstsh7fp2k46juF2fSXNo%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371603954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ITSsd0LE%2BJs12d8nqIjKBgaTwBCxGUfW1dIUNSArF7s%3D&reserved=0>
FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=msZEifJqOREtlckmKcBMN4BAlqBeNq3VzRw%2FdsUH%2BM8%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371608932%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mt%2FzxcINhRfSOan9FnLeEuOCRSaA4iDzVZJS23NgDSA%3D&reserved=0>

______________________________________________________________________________
Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qavJ9f%2B988KYI3EMI3oPi7z%2BWJ6e01Kiphqo8aiLTzg%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371613911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IuaTRgk%2FJM6QVXNGHAQyVgXZDFc6G0JS2RQE%2BNeaYkQ%3D&reserved=0>
Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vNxB%2B2ayi4V8aOF0SG6VCv4QK2TPrZIiwznujhvhOJw%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371613911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wAAPSFdICCMM6ryNUEtCpiGlQSEnhqmrAv12KZ6se4c%3D&reserved=0>
FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=msZEifJqOREtlckmKcBMN4BAlqBeNq3VzRw%2FdsUH%2BM8%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cmark.faine%40nasa.gov%7C1b41867018ff4d9fbb6e08d91c62334e%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637572030371618889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tF88RPhlM7FcLq35HeMiYBGRq0HvtCkoEWy5ceUMZAs%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210521/b24ab4f7/attachment-0001.html>