<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi All,<div><br></div><div>I would like to ask for help with usage of grouping-by function.</div><div><br></div><div>Goal is to correlate logs from auditd. </div><div><br></div><div>Logs produced by auditd are multiline, but events share some number in key with name <b>msg. </b>This in theory should allow aggregation of logs with the same <b>msg </b>key, but I cannot get it right.</div><div><br></div><div>Here is my config:</div><div><br></div><div><div><b>log {</b></div><div><b>  source {</b></div><div><b>    file('/var/log/audit/audit.log' flags(no-parse));</b></div><div><b>  };</b></div><div><b>  parser {</b></div><div><b>    linux-audit-parser(prefix(".auditd."));</b></div><div><b>  };</b></div><div><b>  parser {</b></div><div><b>    grouping-by(</b></div><div><b>      key("${.auditd.msg}")</b></div><div><b>      timeout(10)</b></div><div><b>      aggregate(value("MESSAGE" "TEST"))</b></div><div><b>    );</b></div><div><b>  };</b></div><div><b>  destination {</b></div><div><b>    file('/tmp/test.json' template("$(format-json .auditd.*)\n"));</b></div><div><b>  };</b></div><div><b>};</b></div></div><div><br></div><div>After implementing above config, output logs look like this, prefix is added as expected and format is set o JSON, but no aggregation happens:</div><div><br></div><div><div><b>{"_auditd":{"type":"EXECVE","msg":"audit(1604391099.830:195558):","argc":"2","a1":"-F","a0":"/usr/sbin/syslog-ng"}}</b></div><div><b>{"_auditd":{"type":"CWD","msg":"audit(1604391099.830:195558):","cwd":"/"}}</b></div><div><b>{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/usr/sbin/syslog-ng","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"0","inode":"1078935","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}</b></div><div><b>{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/lib64/ld-linux-x86-64.so.2","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"1","inode":"1048592","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}</b></div><div><b>{"_auditd":{"type":"PROCTITLE","proctitle":"/usr/sbin/syslog-ng\t-F","msg":"audit(1604391099.830:195558):"}}</b></div></div><div><br></div><div>Syslog-ng version I'm using:</div><div><br></div><div><div><b>syslog-ng 3 (3.29.1)</b></div><div><b>Config version: 3.29</b></div><div><b>Installer-Version: 3.29.1</b></div><div><b>Revision: 3.29.1-1</b></div><div><b>Compile-Date: Aug 28 2020 12:13:25</b></div><div><b>Module-Directory: /usr/lib/syslog-ng/3.29</b></div><div><b>Module-Path: /usr/lib/syslog-ng/3.29</b></div><div><b>Include-Path: /usr/share/syslog-ng/include</b></div><div><b>Available-Modules: pseudofile,cef,confgen,stardate,tfgetent,kvformat,afmongodb,afamqp,afsocket,system-source,csvparser,geoip2-plugin,syslogformat,graphite,affile,xml,dbparser,afprog,add-contextual-data,mod-python,sdjournal,map-value-pairs,timestamp,pacctformat,disk-buffer,afstomp,afsmtp,basicfuncs,afuser,tags-parser,linux-kmsg-format,hook-commands,examples,appmodel,afsql,cryptofuncs,json-plugin,riemann,redis</b></div><div><b>Enable-Debug: off</b></div><div><b>Enable-GProf: off</b></div><div><b>Enable-Memtrace: off</b></div><div><b>Enable-IPv6: on</b></div><div><b>Enable-Spoof-Source: on</b></div><div><b>Enable-TCP-Wrapper: on</b></div><div><b>Enable-Linux-Caps: on</b></div><div><b>Enable-Systemd: on</b></div></div><div><br></div><div>Can anyone please tell what am I doing wrong ? </div><div><br></div><div>Kind regards</div><div>Maciej</div></div></div></div></div></div>