Great to see that the Linux Audit parser and grouping-by() is finally getting used. Would be great if you could summarize your use case and the solution in a blog post somewhere. Maybe this could even be published in the syslog-ng.com blog if you don't have anywhere else to post. Cheers, Bazsi On Wed, Nov 4, 2020, 09:56 Maciek Solnicki <msolnicki@gmail.com> wrote:
Hi Gabor,
My use case is this: save every command issued in the Linux system. *Auditd *is a great tool for the job, but it produces logs multiline, which makes them hard to use in external log management solutions. Sure, I could do multiline aggregation on the external system side, but if I would have multiple linux systems sending logs, and from each such system I would need to aggregate multiline logs, it could take a considerable amount of processing power for the log management system. Hence the idea to do it locally, before sending logs out. It also gives me possibility to only forward logs with fields which are of interest to me, skipping everything else:
destination { network( "192.168.1.40" port(10514) log-fifo-size(10000) template("$(format-json *exe cwd a* success pid msg*)\n") );
I can clearly see in the output logs that correlation works as intended in this configuration. Several logs are concatenated into one based on the key *msg*. If I understand correctly it is because of this option: *inherit-mode(context)* which is default. I have set a new value-pair only to see if it works and have something to grep for.
Example from documentation which you provided is not really useful for my use case - in this example, number of correlated messages is known, so you can explicitly state, which fields from which correlated messages are to be copied to output message. But in my case, I do not know how many messages will be correlated, hence I cannot use statements such as *${<field_name>}@<correlated_message_number>.*
But it is just a digression, since my config works as intended, I have no further questions and I thank you all for help :-)
Kind regards Maciej
wt., 3 lis 2020 o 17:34 Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> napisał(a):
Hi Maciek!
I've checked the documentation and I've found documentation bug about an example for value() option:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
Thanks for your notice, I've checked the whole chapter of grouping-by! About improving the documentation: I admit that the chapter where grouping-by() options listed is a bit dense in case of the value() in the aggregate() option. We will discuss this with the doc writer team, when they process the grouping-by() parser example config bug I've reported.
Otherwise, I think Fabien helped you find out where is the problem: in your destination side template, you only include the .auditd. macros, which have been parsed by linux-auditd-parser(). $MESSAGE macro was missing from the template which is set by the grouping-by parser.
I don't know your use case, but I think your current solution lacks any usage of correlation: even though you set a new name-value pair in the aggregated message (.auditd.test), it's basically the same message as the last message that arrived into the same context. As Fabien said, you will see that same message twice (the last message before the timeout expired). The above link shows a good example (I'm copying a fixed version of it) what you can do with message contexts:
aggregate( ... value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}') )
Regards, Gabor
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli <wernli@in2p3.fr> *Sent:* Tuesday, November 3, 2020 15:32 *To:* Maciek Solnicki <msolnicki@gmail.com> *Cc:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Requesting help with Grouping-by function
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi,
If you want to see more Macros in json you can use scopes, for instance:
format-json -s nv-pairs # all generic non-dot macros format-json -s all-nv-pairs # all generic macros format-json -s everything # as advertised
cheers
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq