[syslog-ng] Requesting help with Grouping-by function

Wed Nov 4 13:56:59 UTC 2020

Great to see that the Linux Audit parser and grouping-by() is finally
getting used. Would be great if you could summarize your use case and the
solution in a blog post somewhere.

Maybe this could even be published in the syslog-ng.com blog if you don't
have anywhere else to post.

Cheers,
Bazsi

On Wed, Nov 4, 2020, 09:56 Maciek Solnicki <msolnicki at gmail.com> wrote:

> Hi Gabor,
>
> My use case is this: save every command issued in the Linux system. *Auditd
> *is a great tool for the job, but it produces logs multiline, which makes
> them hard to use in external log management solutions. Sure, I could do
> multiline aggregation on the external system side, but if I would have
> multiple linux systems sending logs, and from each such system I would need
> to aggregate multiline logs, it could take a considerable amount of
> processing power for the log management system. Hence the idea to do it
> locally, before sending logs out. It also gives me possibility to only
> forward logs with fields which are of interest to me, skipping everything
> else:
>
>   destination {
>     network(
>         "192.168.1.40"
>         port(10514)
>         log-fifo-size(10000)
>         template("$(format-json *exe cwd a* success pid msg*)\n")
>     );
>
> I can clearly see in the output logs that correlation works as intended in
> this configuration. Several logs are concatenated into one based on the key
> *msg*. If I understand correctly it is because of this option:
> *inherit-mode(context)* which is default. I have set a new value-pair
> only to see if it works and have something to grep for.
>
> Example from documentation which you provided is not really useful for my
> use case - in this example, number of correlated messages is known, so you
> can explicitly state, which fields from which correlated messages are to be
> copied to output message. But in my case, I do not know how many messages
> will be correlated, hence I cannot use statements such as
> *${<field_name>}@<correlated_message_number>.*
>
> But it is just a digression, since my config works as intended, I have no
> further questions and I thank you all for help :-)
>
> Kind regards
> Maciej
>
> wt., 3 lis 2020 o 17:34 Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
> napisał(a):
>
>> Hi Maciek!
>>
>> I've checked the documentation and I've found  documentation bug about an
>> example for value() option:
>>
>> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/86#TOPIC-1431237
>>
>> Thanks for your notice, I've checked the whole chapter of grouping-by!
>> About improving the documentation:
>> I admit that the chapter where grouping-by() options listed is a bit
>> dense in case of the value() in the aggregate() option.
>> We will discuss this with the doc writer team, when they process the
>> grouping-by() parser example config bug I've reported.
>>
>> Otherwise, I think Fabien helped you find out where is the problem: in
>> your destination side template, you only include the .auditd. macros, which
>> have been parsed by linux-auditd-parser().
>> $MESSAGE macro was missing from the template which is set by the
>> grouping-by parser.
>>
>> I don't know your use case, but I think your current solution lacks any
>> usage of correlation: even though you set a new name-value pair in the
>> aggregated message (.auditd.test), it's basically the same message as the
>> last message that arrived into the same context.
>> As Fabien said, you will see that same message twice (the last message
>> before the timeout expired).
>> The above link shows a good example (I'm copying a fixed version of it)
>> what you can do with message contexts:
>>
>> aggregate(
>>   ...
>>   value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from
>> ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to
>> ${DATE}')
>> )
>>
>> Regards,
>> Gabor
>>
>> ------------------------------
>> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of
>> Fabien Wernli <wernli at in2p3.fr>
>> *Sent:* Tuesday, November 3, 2020 15:32
>> *To:* Maciek Solnicki <msolnicki at gmail.com>
>> *Cc:* Syslog-ng users' and developers' mailing list <
>> syslog-ng at lists.balabit.hu>
>> *Subject:* Re: [syslog-ng] Requesting help with Grouping-by function
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> follow guidance, click links, or open attachments unless you recognize the
>> sender and know the content is safe.
>>
>>
>> Hi,
>>
>> If you want to see more Macros in json you can use scopes, for instance:
>>
>>    format-json -s nv-pairs     # all generic non-dot macros
>>    format-json -s all-nv-pairs # all generic macros
>>    format-json -s everything   # as advertised
>>
>> cheers
>>
>>
>> ______________________________________________________________________________
>> Member info:
>> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0
>> Documentation:
>> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0
>> FAQ:
>> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201104/3bb1b7c8/attachment.html>