[syslog-ng] Requesting help with Grouping-by function

[Maciek Solnicki]

Hi All,

I would like to ask for help with usage of grouping-by function.

Goal is to correlate logs from auditd.

Logs produced by auditd are multiline, but events share some number in key
with name *msg. *This in theory should allow aggregation of logs with the
same *msg *key, but I cannot get it right.

Here is my config:

*log {*
*  source {*
*    file('/var/log/audit/audit.log' flags(no-parse));*
*  };*
*  parser {*
*    linux-audit-parser(prefix(".auditd."));*
*  };*
*  parser {*
*    grouping-by(*
*      key("${.auditd.msg}")*
*      timeout(10)*
*      aggregate(value("MESSAGE" "TEST"))*
*    );*
*  };*
*  destination {*
*    file('/tmp/test.json' template("$(format-json .auditd.*)\n"));*
*  };*
*};*

After implementing above config, output logs look like this, prefix is
added as expected and format is set o JSON, but no aggregation happens:

*{"_auditd":{"type":"EXECVE","msg":"audit(1604391099.830:195558):","argc":"2","a1":"-F","a0":"/usr/sbin/syslog-ng"}}*
*{"_auditd":{"type":"CWD","msg":"audit(1604391099.830:195558):","cwd":"/"}}*
*{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/usr/sbin/syslog-ng","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"0","inode":"1078935","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}*
*{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/lib64/ld-linux-x86-64.so.2","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"1","inode":"1048592","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}*
*{"_auditd":{"type":"PROCTITLE","proctitle":"/usr/sbin/syslog-ng\t-F","msg":"audit(1604391099.830:195558):"}}*

Syslog-ng version I'm using:

*syslog-ng 3 (3.29.1)*
*Config version: 3.29*
*Installer-Version: 3.29.1*
*Revision: 3.29.1-1*
*Compile-Date: Aug 28 2020 12:13:25*
*Module-Directory: /usr/lib/syslog-ng/3.29*
*Module-Path: /usr/lib/syslog-ng/3.29*
*Include-Path: /usr/share/syslog-ng/include*
*Available-Modules:
pseudofile,cef,confgen,stardate,tfgetent,kvformat,afmongodb,afamqp,afsocket,system-source,csvparser,geoip2-plugin,syslogformat,graphite,affile,xml,dbparser,afprog,add-contextual-data,mod-python,sdjournal,map-value-pairs,timestamp,pacctformat,disk-buffer,afstomp,afsmtp,basicfuncs,afuser,tags-parser,linux-kmsg-format,hook-commands,examples,appmodel,afsql,cryptofuncs,json-plugin,riemann,redis*
*Enable-Debug: off*
*Enable-GProf: off*
*Enable-Memtrace: off*
*Enable-IPv6: on*
*Enable-Spoof-Source: on*
*Enable-TCP-Wrapper: on*
*Enable-Linux-Caps: on*
*Enable-Systemd: on*

Can anyone please tell what am I doing wrong ?

Kind regards
Maciej
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201103/66912786/attachment.html>