[syslog-ng] Using custom parser with only a subset of udp traffic

[Peter Czanik (pczanik)]

Hi Syslog-ng,

My blog about analyzing Suricata log messages has many examples about processing logs differently based on message content: https://www.syslog-ng.com/community/b/blog/posts/analyze-your-suricata-logs-in-real-time-using-syslog-ng

Have a nice weekend!
Peter

Peter Czanik (CzP) <peter.czanik at oneidentity.com>
Balabit (a OneIdentity company) / syslog-ng upstream
https://syslog-ng.com/community/
https://twitter.com/PCzanik

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov>
Sent: Friday, May 21, 2021 16:05
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Using custom parser with only a subset of udp traffic

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


I think I have figured out that I can use one source and always specify the parser manually, that has gotten me most of the way there.  Thank you. Still any suggestions for how to proceed are of course welcome.

Thanks,
-Mark

-----Original Message-----
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Faine, Mark R. (MSFC-IS40)[NICS]
Sent: Thursday, May 20, 2021 09:55
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Using custom parser with only a subset of udp traffic

Hopefully one of you can help me figure this out.   All of my udp traffic is coming in through one source (network driver), some of it is firewall data but I have to filter it out based on where it is coming from and other fields in the log data.  I want to use the csv_parser here since it would make it easier for me to get the data I need from the firewall data.  Does this require splitting my one source into two sources, since I would need to use flags('no-parse') on a firewall data source?   I also think this means that I can't use a single log path to process both sources?  Also, how to prevent potential duplication since the non-firewall source will still receive the firewall traffic and try to process it with syslog-parser.

I think the flow is something like this:

                        firewall source -> custom parser -> filters ->rewrites
                      /                                                                                          \
all_data ->                                                                                                 \  -> destination
                    \                                                                                            /
                     all udp source -> filters -> rewrites                             /

I understand this is a bit complex to answer simply, I'm mostly looking for answers to the above questions as well as high level guidance for how to proceed.

Thanks,
-Mark
______________________________________________________________________________
Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476926994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pfdd17XkUlj53ZWtwjloGxs%2BFCVyoSyCWg%2FsMzni%2FJE%3D&reserved=0
Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476926994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZvjFq4y29jjLnicSXcVFbDkstsh7fp2k46juF2fSXNo%3D&reserved=0
FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=msZEifJqOREtlckmKcBMN4BAlqBeNq3VzRw%2FdsUH%2BM8%3D&reserved=0

______________________________________________________________________________
Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qavJ9f%2B988KYI3EMI3oPi7z%2BWJ6e01Kiphqo8aiLTzg%3D&reserved=0
Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vNxB%2B2ayi4V8aOF0SG6VCv4QK2TPrZIiwznujhvhOJw%3D&reserved=0
FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CPeter.Czanik%40oneidentity.com%7Cecfcc63d70fe4cfe5d7e08d91c6186cb%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637572027476936958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=msZEifJqOREtlckmKcBMN4BAlqBeNq3VzRw%2FdsUH%2BM8%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210521/432ac5fc/attachment-0001.html>