[syslog-ng] Requesting help with Grouping-by function

Wed Nov 4 14:19:45 UTC 2020

Hi Bazsi,

I'm creating this use case for a log management product -
https://www.logmanager.cz/en/. I'm a part of the engineering team. We are
constantly on a lookout for ways to improve our solution capabilities, so
this is the reason I'm working on it.

If you do not mind that I'm developing this use case for
commercial purposes, we do have a blog space, so it would be possible to
explain this use case in detail there. Of course the syslog-ng
configuration I've created can still be used with other log management
solutions, not necessarily ours, so such blog post would still be
beneficial to the community.

Kind regards
Maciej

śr., 4 lis 2020 o 14:57 Balazs Scheidler <bazsi77 at gmail.com> napisał(a):

> Great to see that the Linux Audit parser and grouping-by() is finally
> getting used. Would be great if you could summarize your use case and the
> solution in a blog post somewhere.
>
> Maybe this could even be published in the syslog-ng.com blog if you don't
> have anywhere else to post.
>
> Cheers,
> Bazsi
>
>
> On Wed, Nov 4, 2020, 09:56 Maciek Solnicki <msolnicki at gmail.com> wrote:
>
>> Hi Gabor,
>>
>> My use case is this: save every command issued in the Linux system. *Auditd
>> *is a great tool for the job, but it produces logs multiline, which
>> makes them hard to use in external log management solutions. Sure, I could
>> do multiline aggregation on the external system side, but if I would have
>> multiple linux systems sending logs, and from each such system I would need
>> to aggregate multiline logs, it could take a considerable amount of
>> processing power for the log management system. Hence the idea to do it
>> locally, before sending logs out. It also gives me possibility to only
>> forward logs with fields which are of interest to me, skipping everything
>> else:
>>
>>   destination {
>>     network(
>>         "192.168.1.40"
>>         port(10514)
>>         log-fifo-size(10000)
>>         template("$(format-json *exe cwd a* success pid msg*)\n")
>>     );
>>
>> I can clearly see in the output logs that correlation works as intended
>> in this configuration. Several logs are concatenated into one based on the
>> key *msg*. If I understand correctly it is because of this option:
>> *inherit-mode(context)* which is default. I have set a new value-pair
>> only to see if it works and have something to grep for.
>>
>> Example from documentation which you provided is not really useful for my
>> use case - in this example, number of correlated messages is known, so you
>> can explicitly state, which fields from which correlated messages are to be
>> copied to output message. But in my case, I do not know how many messages
>> will be correlated, hence I cannot use statements such as
>> *${<field_name>}@<correlated_message_number>.*
>>
>> But it is just a digression, since my config works as intended, I have no
>> further questions and I thank you all for help :-)
>>
>> Kind regards
>> Maciej
>>
>> wt., 3 lis 2020 o 17:34 Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
>> napisał(a):
>>
>>> Hi Maciek!
>>>
>>> I've checked the documentation and I've found  documentation bug about
>>> an example for value() option:
>>>
>>> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/86#TOPIC-1431237
>>>
>>> Thanks for your notice, I've checked the whole chapter of grouping-by!
>>> About improving the documentation:
>>> I admit that the chapter where grouping-by() options listed is a bit
>>> dense in case of the value() in the aggregate() option.
>>> We will discuss this with the doc writer team, when they process the
>>> grouping-by() parser example config bug I've reported.
>>>
>>> Otherwise, I think Fabien helped you find out where is the problem: in
>>> your destination side template, you only include the .auditd. macros, which
>>> have been parsed by linux-auditd-parser().
>>> $MESSAGE macro was missing from the template which is set by the
>>> grouping-by parser.
>>>
>>> I don't know your use case, but I think your current solution lacks any
>>> usage of correlation: even though you set a new name-value pair in the
>>> aggregated message (.auditd.test), it's basically the same message as the
>>> last message that arrived into the same context.
>>> As Fabien said, you will see that same message twice (the last message
>>> before the timeout expired).
>>> The above link shows a good example (I'm copying a fixed version of it)
>>> what you can do with message contexts:
>>>
>>> aggregate(
>>>   ...
>>>   value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from
>>> ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to
>>> ${DATE}')
>>> )
>>>
>>> Regards,
>>> Gabor
>>>
>>> ------------------------------
>>> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of
>>> Fabien Wernli <wernli at in2p3.fr>
>>> *Sent:* Tuesday, November 3, 2020 15:32
>>> *To:* Maciek Solnicki <msolnicki at gmail.com>
>>> *Cc:* Syslog-ng users' and developers' mailing list <
>>> syslog-ng at lists.balabit.hu>
>>> *Subject:* Re: [syslog-ng] Requesting help with Grouping-by function
>>>
>>> CAUTION: This email originated from outside of the organization. Do not
>>> follow guidance, click links, or open attachments unless you recognize the
>>> sender and know the content is safe.
>>>
>>>
>>> Hi,
>>>
>>> If you want to see more Macros in json you can use scopes, for instance:
>>>
>>>    format-json -s nv-pairs     # all generic non-dot macros
>>>    format-json -s all-nv-pairs # all generic macros
>>>    format-json -s everything   # as advertised
>>>
>>> cheers
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info:
>>> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0
>>> Documentation:
>>> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0
>>> FAQ:
>>> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201104/f3a4cb0c/attachment-0001.html>