Hi All,

I would like to ask for help with usage of grouping-by function.

Goal is to correlate logs from auditd.

Logs produced by auditd are multiline, but events share some number in key with name msg. This in theory should allow aggregation of logs with the same msg key, but I cannot get it right.

Here is my config:

log {

source {

file('/var/log/audit/audit.log' flags(no-parse));

};

parser {

linux-audit-parser(prefix(".auditd."));

};

parser {

grouping-by(

key("${.auditd.msg}")

timeout(10)

aggregate(value("MESSAGE" "TEST"))

);

};

destination {

file('/tmp/test.json' template("$(format-json .auditd.*)\n"));

};

After implementing above config, output logs look like this, prefix is added as expected and format is set o JSON, but no aggregation happens:

{"_auditd":{"type":"EXECVE","msg":"audit(1604391099.830:195558):","argc":"2","a1":"-F","a0":"/usr/sbin/syslog-ng"}}

{"_auditd":{"type":"CWD","msg":"audit(1604391099.830:195558):","cwd":"/"}}

{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/usr/sbin/syslog-ng","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"0","inode":"1078935","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}

{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/lib64/ld-linux-x86-64.so.2","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"1","inode":"1048592","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}

{"_auditd":{"type":"PROCTITLE","proctitle":"/usr/sbin/syslog-ng\t-F","msg":"audit(1604391099.830:195558):"}}

Syslog-ng version I'm using:

syslog-ng 3 (3.29.1)

Config version: 3.29

Installer-Version: 3.29.1

Revision: 3.29.1-1

Compile-Date: Aug 28 2020 12:13:25

Module-Directory: /usr/lib/syslog-ng/3.29

Module-Path: /usr/lib/syslog-ng/3.29

Include-Path: /usr/share/syslog-ng/include

Available-Modules: pseudofile,cef,confgen,stardate,tfgetent,kvformat,afmongodb,afamqp,afsocket,system-source,csvparser,geoip2-plugin,syslogformat,graphite,affile,xml,dbparser,afprog,add-contextual-data,mod-python,sdjournal,map-value-pairs,timestamp,pacctformat,disk-buffer,afstomp,afsmtp,basicfuncs,afuser,tags-parser,linux-kmsg-format,hook-commands,examples,appmodel,afsql,cryptofuncs,json-plugin,riemann,redis

Enable-Debug: off

Enable-GProf: off

Enable-Memtrace: off

Enable-IPv6: on

Enable-Spoof-Source: on

Enable-TCP-Wrapper: on

Enable-Linux-Caps: on

Enable-Systemd: on

Can anyone please tell what am I doing wrong ?

Kind regards

Maciej