Requesting help with Grouping-by function
Hi All, I would like to ask for help with usage of grouping-by function. Goal is to correlate logs from auditd. Logs produced by auditd are multiline, but events share some number in key with name *msg. *This in theory should allow aggregation of logs with the same *msg *key, but I cannot get it right. Here is my config: *log {* * source {* * file('/var/log/audit/audit.log' flags(no-parse));* * };* * parser {* * linux-audit-parser(prefix(".auditd."));* * };* * parser {* * grouping-by(* * key("${.auditd.msg}")* * timeout(10)* * aggregate(value("MESSAGE" "TEST"))* * );* * };* * destination {* * file('/tmp/test.json' template("$(format-json .auditd.*)\n"));* * };* *};* After implementing above config, output logs look like this, prefix is added as expected and format is set o JSON, but no aggregation happens: *{"_auditd":{"type":"EXECVE","msg":"audit(1604391099.830:195558):","argc":"2","a1":"-F","a0":"/usr/sbin/syslog-ng"}}* *{"_auditd":{"type":"CWD","msg":"audit(1604391099.830:195558):","cwd":"/"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/usr/sbin/syslog-ng","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"0","inode":"1078935","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/lib64/ld-linux-x86-64.so.2","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"1","inode":"1048592","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PROCTITLE","proctitle":"/usr/sbin/syslog-ng\t-F","msg":"audit(1604391099.830:195558):"}}* Syslog-ng version I'm using: *syslog-ng 3 (3.29.1)* *Config version: 3.29* *Installer-Version: 3.29.1* *Revision: 3.29.1-1* *Compile-Date: Aug 28 2020 12:13:25* *Module-Directory: /usr/lib/syslog-ng/3.29* *Module-Path: /usr/lib/syslog-ng/3.29* *Include-Path: /usr/share/syslog-ng/include* *Available-Modules: pseudofile,cef,confgen,stardate,tfgetent,kvformat,afmongodb,afamqp,afsocket,system-source,csvparser,geoip2-plugin,syslogformat,graphite,affile,xml,dbparser,afprog,add-contextual-data,mod-python,sdjournal,map-value-pairs,timestamp,pacctformat,disk-buffer,afstomp,afsmtp,basicfuncs,afuser,tags-parser,linux-kmsg-format,hook-commands,examples,appmodel,afsql,cryptofuncs,json-plugin,riemann,redis* *Enable-Debug: off* *Enable-GProf: off* *Enable-Memtrace: off* *Enable-IPv6: on* *Enable-Spoof-Source: on* *Enable-TCP-Wrapper: on* *Enable-Linux-Caps: on* *Enable-Systemd: on* Can anyone please tell what am I doing wrong ? Kind regards Maciej
Hi Maciek, On Tue, Nov 03, 2020 at 12:24:40PM +0100, Maciek Solnicki wrote:
* destination {* * file('/tmp/test.json' template("$(format-json .auditd.*)\n"));* * };* *};*
This means you're outputting the contents of all `.auditd.` macros to file test.json. But your grouping-by parser generates a message with the macro MESSAGE set to the value "TEST" :
* grouping-by(* * key("${.auditd.msg}")* * timeout(10)* * aggregate(value("MESSAGE" "TEST"))* * );*
So you won't see it in test.json. Well, more exactly you should see it as the default value of aggregate(inherit-mode()) is "context", but you won't see the MESSAGE:TEST macro as you're omitting it from your output. I'm guessing you should see a message twice in that case.
Hi Fabien, Thanks for your response. I was using this snippet of code (taken from documentation) before, but doesn't seem to work either: *value("MESSAGE" "$(format-json .auditd.*)")* So how do I output an aggregated message to the test.json file ? Or any file. I apologize if my questions are basic, but I had a hard time finding answers in documentation. Grouping-by() function could definitely use more explanation. Kind regards Maciej wt., 3 lis 2020 o 13:05 Fabien Wernli <wernli@in2p3.fr> napisał(a):
Hi Maciek,
On Tue, Nov 03, 2020 at 12:24:40PM +0100, Maciek Solnicki wrote:
* destination {* * file('/tmp/test.json' template("$(format-json .auditd.*)\n"));* * };* *};*
This means you're outputting the contents of all `.auditd.` macros to file test.json.
But your grouping-by parser generates a message with the macro MESSAGE set to the value "TEST" :
* grouping-by(* * key("${.auditd.msg}")* * timeout(10)* * aggregate(value("MESSAGE" "TEST"))* * );*
So you won't see it in test.json. Well, more exactly you should see it as the default value of aggregate(inherit-mode()) is "context", but you won't see the MESSAGE:TEST macro as you're omitting it from your output. I'm guessing you should see a message twice in that case.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Nevermind! I think I got it :-) *log {* * source {* * file('/var/log/audit/audit.log' flags(no-parse));* * };* * parser {* * linux-audit-parser(prefix(".auditd."));* * };* * parser {* * grouping-by(* * key("${.auditd.msg}")* * aggregate(* * value(".auditd.test" "#########TEST")* * )* * timeout(10)* * );* * };* * destination {* * file('/tmp/test.json' template("$(format-json .auditd.exe .auditd.cwd .auditd.a* .auditd.test .auditd.success .auditd.pid .auditd.msg)\n"));* * };* *};* This seems to work as now result looks like this: *{* * "_auditd": {* * "test": "#########TEST",* * "success": "yes",* * "pid": "57483",* * "msg": "audit(1604412718.657:211575):",* * "exe": "/usr/bin/jq",* * "cwd": "/etc/syslog-ng/conf.d",* * "auid": "1000",* * "argc": "1",* * "arch": "c000003e",* * "a3": "fffffffffffff878",* * "a2": "564235a17dd0",* * "a1": "564235a20680",* * "a0": "jq"* * }* *}* If you have any improvements ideas please let me know. Thanks! Maciej wt., 3 lis 2020 o 14:30 Maciek Solnicki <msolnicki@gmail.com> napisał(a):
Hi Fabien,
Thanks for your response.
I was using this snippet of code (taken from documentation) before, but doesn't seem to work either:
*value("MESSAGE" "$(format-json .auditd.*)")*
So how do I output an aggregated message to the test.json file ? Or any file.
I apologize if my questions are basic, but I had a hard time finding answers in documentation. Grouping-by() function could definitely use more explanation.
Kind regards Maciej
wt., 3 lis 2020 o 13:05 Fabien Wernli <wernli@in2p3.fr> napisał(a):
Hi Maciek,
On Tue, Nov 03, 2020 at 12:24:40PM +0100, Maciek Solnicki wrote:
* destination {* * file('/tmp/test.json' template("$(format-json .auditd.*)\n"));* * };* *};*
This means you're outputting the contents of all `.auditd.` macros to file test.json.
But your grouping-by parser generates a message with the macro MESSAGE set to the value "TEST" :
* grouping-by(* * key("${.auditd.msg}")* * timeout(10)* * aggregate(value("MESSAGE" "TEST"))* * );*
So you won't see it in test.json. Well, more exactly you should see it as the default value of aggregate(inherit-mode()) is "context", but you won't see the MESSAGE:TEST macro as you're omitting it from your output. I'm guessing you should see a message twice in that case.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, If you want to see more Macros in json you can use scopes, for instance: format-json -s nv-pairs # all generic non-dot macros format-json -s all-nv-pairs # all generic macros format-json -s everything # as advertised cheers
Hi Maciek! I've checked the documentation and I've found documentation bug about an example for value() option: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... Thanks for your notice, I've checked the whole chapter of grouping-by! About improving the documentation: I admit that the chapter where grouping-by() options listed is a bit dense in case of the value() in the aggregate() option. We will discuss this with the doc writer team, when they process the grouping-by() parser example config bug I've reported. Otherwise, I think Fabien helped you find out where is the problem: in your destination side template, you only include the .auditd. macros, which have been parsed by linux-auditd-parser(). $MESSAGE macro was missing from the template which is set by the grouping-by parser. I don't know your use case, but I think your current solution lacks any usage of correlation: even though you set a new name-value pair in the aggregated message (.auditd.test), it's basically the same message as the last message that arrived into the same context. As Fabien said, you will see that same message twice (the last message before the timeout expired). The above link shows a good example (I'm copying a fixed version of it) what you can do with message contexts: aggregate( ... value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}') ) Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli <wernli@in2p3.fr> Sent: Tuesday, November 3, 2020 15:32 To: Maciek Solnicki <msolnicki@gmail.com> Cc: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Requesting help with Grouping-by function CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, If you want to see more Macros in json you can use scopes, for instance: format-json -s nv-pairs # all generic non-dot macros format-json -s all-nv-pairs # all generic macros format-json -s everything # as advertised cheers ______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
Hi Gabor, My use case is this: save every command issued in the Linux system. *Auditd *is a great tool for the job, but it produces logs multiline, which makes them hard to use in external log management solutions. Sure, I could do multiline aggregation on the external system side, but if I would have multiple linux systems sending logs, and from each such system I would need to aggregate multiline logs, it could take a considerable amount of processing power for the log management system. Hence the idea to do it locally, before sending logs out. It also gives me possibility to only forward logs with fields which are of interest to me, skipping everything else: destination { network( "192.168.1.40" port(10514) log-fifo-size(10000) template("$(format-json *exe cwd a* success pid msg*)\n") ); I can clearly see in the output logs that correlation works as intended in this configuration. Several logs are concatenated into one based on the key *msg*. If I understand correctly it is because of this option: *inherit-mode(context)* which is default. I have set a new value-pair only to see if it works and have something to grep for. Example from documentation which you provided is not really useful for my use case - in this example, number of correlated messages is known, so you can explicitly state, which fields from which correlated messages are to be copied to output message. But in my case, I do not know how many messages will be correlated, hence I cannot use statements such as *${<field_name>}@<correlated_message_number>.* But it is just a digression, since my config works as intended, I have no further questions and I thank you all for help :-) Kind regards Maciej wt., 3 lis 2020 o 17:34 Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> napisał(a):
Hi Maciek!
I've checked the documentation and I've found documentation bug about an example for value() option:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
Thanks for your notice, I've checked the whole chapter of grouping-by! About improving the documentation: I admit that the chapter where grouping-by() options listed is a bit dense in case of the value() in the aggregate() option. We will discuss this with the doc writer team, when they process the grouping-by() parser example config bug I've reported.
Otherwise, I think Fabien helped you find out where is the problem: in your destination side template, you only include the .auditd. macros, which have been parsed by linux-auditd-parser(). $MESSAGE macro was missing from the template which is set by the grouping-by parser.
I don't know your use case, but I think your current solution lacks any usage of correlation: even though you set a new name-value pair in the aggregated message (.auditd.test), it's basically the same message as the last message that arrived into the same context. As Fabien said, you will see that same message twice (the last message before the timeout expired). The above link shows a good example (I'm copying a fixed version of it) what you can do with message contexts:
aggregate( ... value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}') )
Regards, Gabor
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli <wernli@in2p3.fr> *Sent:* Tuesday, November 3, 2020 15:32 *To:* Maciek Solnicki <msolnicki@gmail.com> *Cc:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Requesting help with Grouping-by function
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi,
If you want to see more Macros in json you can use scopes, for instance:
format-json -s nv-pairs # all generic non-dot macros format-json -s all-nv-pairs # all generic macros format-json -s everything # as advertised
cheers
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
Great to see that the Linux Audit parser and grouping-by() is finally getting used. Would be great if you could summarize your use case and the solution in a blog post somewhere. Maybe this could even be published in the syslog-ng.com blog if you don't have anywhere else to post. Cheers, Bazsi On Wed, Nov 4, 2020, 09:56 Maciek Solnicki <msolnicki@gmail.com> wrote:
Hi Gabor,
My use case is this: save every command issued in the Linux system. *Auditd *is a great tool for the job, but it produces logs multiline, which makes them hard to use in external log management solutions. Sure, I could do multiline aggregation on the external system side, but if I would have multiple linux systems sending logs, and from each such system I would need to aggregate multiline logs, it could take a considerable amount of processing power for the log management system. Hence the idea to do it locally, before sending logs out. It also gives me possibility to only forward logs with fields which are of interest to me, skipping everything else:
destination { network( "192.168.1.40" port(10514) log-fifo-size(10000) template("$(format-json *exe cwd a* success pid msg*)\n") );
I can clearly see in the output logs that correlation works as intended in this configuration. Several logs are concatenated into one based on the key *msg*. If I understand correctly it is because of this option: *inherit-mode(context)* which is default. I have set a new value-pair only to see if it works and have something to grep for.
Example from documentation which you provided is not really useful for my use case - in this example, number of correlated messages is known, so you can explicitly state, which fields from which correlated messages are to be copied to output message. But in my case, I do not know how many messages will be correlated, hence I cannot use statements such as *${<field_name>}@<correlated_message_number>.*
But it is just a digression, since my config works as intended, I have no further questions and I thank you all for help :-)
Kind regards Maciej
wt., 3 lis 2020 o 17:34 Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> napisał(a):
Hi Maciek!
I've checked the documentation and I've found documentation bug about an example for value() option:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
Thanks for your notice, I've checked the whole chapter of grouping-by! About improving the documentation: I admit that the chapter where grouping-by() options listed is a bit dense in case of the value() in the aggregate() option. We will discuss this with the doc writer team, when they process the grouping-by() parser example config bug I've reported.
Otherwise, I think Fabien helped you find out where is the problem: in your destination side template, you only include the .auditd. macros, which have been parsed by linux-auditd-parser(). $MESSAGE macro was missing from the template which is set by the grouping-by parser.
I don't know your use case, but I think your current solution lacks any usage of correlation: even though you set a new name-value pair in the aggregated message (.auditd.test), it's basically the same message as the last message that arrived into the same context. As Fabien said, you will see that same message twice (the last message before the timeout expired). The above link shows a good example (I'm copying a fixed version of it) what you can do with message contexts:
aggregate( ... value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}') )
Regards, Gabor
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli <wernli@in2p3.fr> *Sent:* Tuesday, November 3, 2020 15:32 *To:* Maciek Solnicki <msolnicki@gmail.com> *Cc:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Requesting help with Grouping-by function
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi,
If you want to see more Macros in json you can use scopes, for instance:
format-json -s nv-pairs # all generic non-dot macros format-json -s all-nv-pairs # all generic macros format-json -s everything # as advertised
cheers
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Bazsi, I'm creating this use case for a log management product - https://www.logmanager.cz/en/. I'm a part of the engineering team. We are constantly on a lookout for ways to improve our solution capabilities, so this is the reason I'm working on it. If you do not mind that I'm developing this use case for commercial purposes, we do have a blog space, so it would be possible to explain this use case in detail there. Of course the syslog-ng configuration I've created can still be used with other log management solutions, not necessarily ours, so such blog post would still be beneficial to the community. Kind regards Maciej śr., 4 lis 2020 o 14:57 Balazs Scheidler <bazsi77@gmail.com> napisał(a):
Great to see that the Linux Audit parser and grouping-by() is finally getting used. Would be great if you could summarize your use case and the solution in a blog post somewhere.
Maybe this could even be published in the syslog-ng.com blog if you don't have anywhere else to post.
Cheers, Bazsi
On Wed, Nov 4, 2020, 09:56 Maciek Solnicki <msolnicki@gmail.com> wrote:
Hi Gabor,
My use case is this: save every command issued in the Linux system. *Auditd *is a great tool for the job, but it produces logs multiline, which makes them hard to use in external log management solutions. Sure, I could do multiline aggregation on the external system side, but if I would have multiple linux systems sending logs, and from each such system I would need to aggregate multiline logs, it could take a considerable amount of processing power for the log management system. Hence the idea to do it locally, before sending logs out. It also gives me possibility to only forward logs with fields which are of interest to me, skipping everything else:
destination { network( "192.168.1.40" port(10514) log-fifo-size(10000) template("$(format-json *exe cwd a* success pid msg*)\n") );
I can clearly see in the output logs that correlation works as intended in this configuration. Several logs are concatenated into one based on the key *msg*. If I understand correctly it is because of this option: *inherit-mode(context)* which is default. I have set a new value-pair only to see if it works and have something to grep for.
Example from documentation which you provided is not really useful for my use case - in this example, number of correlated messages is known, so you can explicitly state, which fields from which correlated messages are to be copied to output message. But in my case, I do not know how many messages will be correlated, hence I cannot use statements such as *${<field_name>}@<correlated_message_number>.*
But it is just a digression, since my config works as intended, I have no further questions and I thank you all for help :-)
Kind regards Maciej
wt., 3 lis 2020 o 17:34 Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> napisał(a):
Hi Maciek!
I've checked the documentation and I've found documentation bug about an example for value() option:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
Thanks for your notice, I've checked the whole chapter of grouping-by! About improving the documentation: I admit that the chapter where grouping-by() options listed is a bit dense in case of the value() in the aggregate() option. We will discuss this with the doc writer team, when they process the grouping-by() parser example config bug I've reported.
Otherwise, I think Fabien helped you find out where is the problem: in your destination side template, you only include the .auditd. macros, which have been parsed by linux-auditd-parser(). $MESSAGE macro was missing from the template which is set by the grouping-by parser.
I don't know your use case, but I think your current solution lacks any usage of correlation: even though you set a new name-value pair in the aggregated message (.auditd.test), it's basically the same message as the last message that arrived into the same context. As Fabien said, you will see that same message twice (the last message before the timeout expired). The above link shows a good example (I'm copying a fixed version of it) what you can do with message contexts:
aggregate( ... value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}') )
Regards, Gabor
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli <wernli@in2p3.fr> *Sent:* Tuesday, November 3, 2020 15:32 *To:* Maciek Solnicki <msolnicki@gmail.com> *Cc:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Requesting help with Grouping-by function
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi,
If you want to see more Macros in json you can use scopes, for instance:
format-json -s nv-pairs # all generic non-dot macros format-json -s all-nv-pairs # all generic macros format-json -s everything # as advertised
cheers
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balazs Scheidler
-
Fabien Wernli
-
Gabor Nagy (gnagy)
-
Maciek Solnicki