[syslog-ng] Requesting help with Grouping-by function

Wed Nov 4 08:56:06 UTC 2020

Hi Gabor,

My use case is this: save every command issued in the Linux system. *Auditd
*is a great tool for the job, but it produces logs multiline, which makes
them hard to use in external log management solutions. Sure, I could do
multiline aggregation on the external system side, but if I would have
multiple linux systems sending logs, and from each such system I would need
to aggregate multiline logs, it could take a considerable amount of
processing power for the log management system. Hence the idea to do it
locally, before sending logs out. It also gives me possibility to only
forward logs with fields which are of interest to me, skipping everything
else:

  destination {
    network(
        "192.168.1.40"
        port(10514)
        log-fifo-size(10000)
        template("$(format-json *exe cwd a* success pid msg*)\n")
    );

I can clearly see in the output logs that correlation works as intended in
this configuration. Several logs are concatenated into one based on the key
*msg*. If I understand correctly it is because of this option:
*inherit-mode(context)* which is default. I have set a new value-pair only
to see if it works and have something to grep for.

Example from documentation which you provided is not really useful for my
use case - in this example, number of correlated messages is known, so you
can explicitly state, which fields from which correlated messages are to be
copied to output message. But in my case, I do not know how many messages
will be correlated, hence I cannot use statements such as
*${<field_name>}@<correlated_message_number>.*

But it is just a digression, since my config works as intended, I have no
further questions and I thank you all for help :-)

Kind regards
Maciej

wt., 3 lis 2020 o 17:34 Gabor Nagy (gnagy) <Gabor.Nagy at oneidentity.com>
napisał(a):

> Hi Maciek!
>
> I've checked the documentation and I've found  documentation bug about an
> example for value() option:
>
> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/86#TOPIC-1431237
>
> Thanks for your notice, I've checked the whole chapter of grouping-by!
> About improving the documentation:
> I admit that the chapter where grouping-by() options listed is a bit dense
> in case of the value() in the aggregate() option.
> We will discuss this with the doc writer team, when they process the
> grouping-by() parser example config bug I've reported.
>
> Otherwise, I think Fabien helped you find out where is the problem: in
> your destination side template, you only include the .auditd. macros, which
> have been parsed by linux-auditd-parser().
> $MESSAGE macro was missing from the template which is set by the
> grouping-by parser.
>
> I don't know your use case, but I think your current solution lacks any
> usage of correlation: even though you set a new name-value pair in the
> aggregated message (.auditd.test), it's basically the same message as the
> last message that arrived into the same context.
> As Fabien said, you will see that same message twice (the last message
> before the timeout expired).
> The above link shows a good example (I'm copying a fixed version of it)
> what you can do with message contexts:
>
> aggregate(
>   ...
>   value('MESSAGE' 'An SSH session for ${SSH_USERNAME}@1 from
> ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}')
> )
>
> Regards,
> Gabor
>
> ------------------------------
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of
> Fabien Wernli <wernli at in2p3.fr>
> *Sent:* Tuesday, November 3, 2020 15:32
> *To:* Maciek Solnicki <msolnicki at gmail.com>
> *Cc:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Requesting help with Grouping-by function
>
> CAUTION: This email originated from outside of the organization. Do not
> follow guidance, click links, or open attachments unless you recognize the
> sender and know the content is safe.
>
>
> Hi,
>
> If you want to see more Macros in json you can use scopes, for instance:
>
>    format-json -s nv-pairs     # all generic non-dot macros
>    format-json -s all-nv-pairs # all generic macros
>    format-json -s everything   # as advertised
>
> cheers
>
>
> ______________________________________________________________________________
> Member info:
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qHmBE%2BaE5kDGmXbGg2E1KdJCKmHp1%2Bw62Uy7BupqlLM%3D&reserved=0
> Documentation:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l0Y1NnxYxEyQstr7N%2Bp%2BKkLTIMvUt6ATDec8B18ufZo%3D&reserved=0
> FAQ:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C5fb2a20040a54610eba508d8800559da%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637400107786812381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dp%2FGkudcz87cnFnuj12bSKjc4TZP1YsoWZnHi1uHmXE%3D&reserved=0
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201104/857b3bfc/attachment.html>