Hello, I'm new to syslog-ng, and I'm having some trouble. I want to use syslog-ng to aggregate similar events from our firewalls. I use the grouping-by parser to create an aggregated event, but I have a question, how do I get only aggregated events on the output without source ones? My configuration: parser p_patterndb { db-parser(file("/etc/syslog-ng/conf.d/forti.xml")); }; log { source { file("/tmp/in.log" flags(no-parse)); }; parser { kv_parser (prefix("forti.")); grouping-by( key("${forti.devname}${forti.srcip}${forti.dstip}${forti.dstport}") scope("process") timeout(15) aggregate( value("MESSAGE" "aggegated devname=${forti.devname} srcip=${forti.srcip} dstip=${forti.dstip} dstport=${forti.dstport} count=$(context-length)") inherit-mode("context") ) ); }; destination { file("tmp/out.log"); }; }; My current output: Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="fw01" devid=" " vd="" date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="-fw01" devid="" vd=" " date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 What I want: Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 Any guidance would be very much appreciated. Maxim Gazin | Senior security engineer | Rostelecom
Hello, Maybe there is a better option, but what I ended up doing in similuar case to set a new key-value and filter messages that had that value. aggregate( ... value("should-only-pass" "1") ... ) and before destination add a filter to only pass messages with should-only-pass -- kokan Get Outlook for Android<https://aka.ms/ghei36> ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Газин Максим Алексеевич <Maksim.Gazin@rt.ru> Sent: Tuesday, June 11, 2019 5:03:02 PM To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] grouping-by question CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello, I'm new to syslog-ng, and I'm having some trouble. I want to use syslog-ng to aggregate similar events from our firewalls. I use the grouping-by parser to create an aggregated event, but I have a question, how do I get only aggregated events on the output without source ones? My configuration: parser p_patterndb { db-parser(file("/etc/syslog-ng/conf.d/forti.xml")); }; log { source { file("/tmp/in.log" flags(no-parse)); }; parser { kv_parser (prefix("forti.")); grouping-by( key("${forti.devname}${forti.srcip}${forti.dstip}${forti.dstport}") scope("process") timeout(15) aggregate( value("MESSAGE" "aggegated devname=${forti.devname} srcip=${forti.srcip} dstip=${forti.dstip} dstport=${forti.dstport} count=$(context-length)") inherit-mode("context") ) ); }; destination { file("tmp/out.log"); }; }; My current output: Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="fw01" devid=" " vd="" date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="-fw01" devid="" vd=" " date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 What I want: Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 Any guidance would be very much appreciated. Maxim Gazin | Senior security engineer | Rostelecom
Thanks for help! This solution helped me. I have one more question: I send several messages that must be aggregated into one, but the context-length parameter always turns out to be increased by one. Why it happens? From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Peter Kokai (pkokai) Sent: Tuesday, June 11, 2019 6:14 PM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] grouping-by question Hello, Maybe there is a better option, but what I ended up doing in similuar case to set a new key-value and filter messages that had that value. aggregate( ... value("should-only-pass" "1") ... ) and before destination add a filter to only pass messages with should-only-pass -- kokan Get Outlook for Android<https://aka.ms/ghei36> ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Газин Максим Алексеевич <Maksim.Gazin@rt.ru<mailto:Maksim.Gazin@rt.ru>> Sent: Tuesday, June 11, 2019 5:03:02 PM To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] grouping-by question CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello, I'm new to syslog-ng, and I'm having some trouble. I want to use syslog-ng to aggregate similar events from our firewalls. I use the grouping-by parser to create an aggregated event, but I have a question, how do I get only aggregated events on the output without source ones? My configuration: parser p_patterndb { db-parser(file("/etc/syslog-ng/conf.d/forti.xml")); }; log { source { file("/tmp/in.log" flags(no-parse)); }; parser { kv_parser (prefix("forti.")); grouping-by( key("${forti.devname}${forti.srcip}${forti.dstip}${forti.dstport}") scope("process") timeout(15) aggregate( value("MESSAGE" "aggegated devname=${forti.devname} srcip=${forti.srcip} dstip=${forti.dstip} dstport=${forti.dstport} count=$(context-length)") inherit-mode("context") ) ); }; destination { file("tmp/out.log"); }; }; My current output: Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="fw01" devid=" " vd="" date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="-fw01" devid="" vd=" " date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 What I want: Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 Any guidance would be very much appreciated. Maxim Gazin | Senior security engineer | Rostelecom
Hi, On Thu, Jun 13, 2019 at 07:35:04AM +0000, Газин Максим Алексеевич wrote:
I have one more question: I send several messages that must be aggregated into one, but the context-length parameter always turns out to be increased by one. Why it happens?
Probably because the timeout is never reached?
what does it mean? Is it possible to make the context-length equal to the value of the aggregated messages? -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Fabien Wernli Sent: Thursday, June 13, 2019 10:42 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] grouping-by question Hi, On Thu, Jun 13, 2019 at 07:35:04AM +0000, Газин Максим Алексеевич wrote:
I have one more question: I send several messages that must be aggregated into one, but the context-length parameter always turns out to be increased by one. Why it happens?
Probably because the timeout is never reached? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Thu, Jun 20, 2019 at 07:17:28AM +0000, Газин Максим Алексеевич wrote:
what does it mean? Is it possible to make the context-length equal to the value of the aggregated messages?
In your original email, your config indicated that you were using `timeout(15)` in the grouping-by config. As far as I understand the grouping-by parser, this means that the context will only be closed (and thus context-len reinitialized) if no message matches the condition for 15 seconds.
participants (3)
-
Fabien Wernli
-
Peter Kokai (pkokai)
-
Газин Максим Алексеевич