[syslog-ng] group-by() send all messages to destination() ?

[Jason Cooper]

Hi all,

On Thu, Oct 03, 2019 at 02:40:16PM +0000, Jason Cooper wrote:
> On Wed, Oct 02, 2019 at 01:33:22PM +0000, Gabor Nagy (gnagy) wrote:
> > What Fabien suggested just rang a bell for me, but wouldn't
> > "context-values" or "context-lookup" template functions  suit your
> > need?
> > 
> > I have copied the admin guide's referring part:
> > 
> > context-values
> > Syntax:
> > $(context-values $name-value1 $name-value2 ...)
> > Description: The context-values template function returns a list of
> > every occurrence of the specified name-value pairs from the entire
> > context. For example, if the context contains multiple messages, the
> > $(context-values ${HOST}) template function will return a comma-
> > separated list of the ${HOST} values that appear in the context.
> 
> Hot damn!  I think so!  It looks like I'll need `implode()` as well to
> string together the messages from `context-values()` separated by
> '\n'...

Well, I'm close :) Here's a dump to a log file destination for a single
grouping-by() match:

```








worker-dev[52030324bc41cf34] IAD:
    2019-10-04T00:24:20.864Z INFO Version: v0.12-4-g4ff99a0cb938
    2019-10-04T00:24:20.864Z INFO POST api-dev.example.com/v1/verifyReceipt called by: WWW.XXX.YYY.ZZZ
    2019-10-04T00:24:21.284Z DEBUG Receipt unchanged since last verified
    2019-10-04T00:24:26.227Z CRIT TypeError: Cannot read property 'duration_ms' of undefined
    2019-10-04T00:24:26.227Z CRIT     at updateProfileShopify (worker.js:698:71)
    2019-10-04T00:24:26.227Z CRIT     at async buildProfile (worker.js:583:15)
    2019-10-04T00:24:26.227Z CRIT     at async verifyReceipt (worker.js:1310:23)
    2019-10-04T00:24:26.227Z CRIT     at async failsafe (worker.js:65:24)
    2019-10-04T00:24:26.227Z INFO BAIL(500) ERROR: Internal server error
    2019-10-04T00:24:26.227Z INFO BAIL(500) ERROR: Internal server error

```

The empty lines above are deliberately retained.  So, I get 8 empty lines,
then my expected output, with the last line inexplicably doubled, and
then one extra empty line.  Clearly I don't have this mastered yet.  :-)

A possible coincidence is that there are 8 non-duplicated messages for 8
empty lines.

Here's the relevant portions of my syslog-ng config:

```
@version: 3.22

# common
parser nginx-lua-parser {
	json-parser (prefix(".json."));
};

parser alert-parser {
   grouping-by(
     key("${.json.rayid}")
     having( "${.json.level}" == "CRIT" )
     trigger(match("BAIL" value(".json.message")))
     aggregate(
       value("MSGS" "${.json.script}[${.json.rayid}] ${.json.colo}:\n$(implode '\n' $(context-lookup ('x' == 'x') $(implode ' ' '   ' ${.json.timestamp} ${.json.level} ${.json.message})))")
       inherit-mode("context")
     )
     inject-mode("pass-through")
     timeout(10)
   );
};

template alert-template "${MSGS}\n";

source worker-src {
	unix-stream("/var/run/nginx-lua/worker.sock", group(nginx) flags(no-parse));
};

# development logs
filter worker-dev-filter {match("-dev" value (".json.script"));};

destination worker-dev-alert-dest {
	file("/var/log/worker/alert-dev.log" template(alert-template));
};

log {
	source(worker-src);
	parser(nginx-lua-parser);
	filter(worker-dev-filter);
	parser(alert-parser);
	destination(worker-dev-alert-dest);
};
```

So, why I am getting empty lines?  and why is the trigger line
duplicated?

Thanks!

Jason.