[syslog-ng] group-by() send all messages to destination() ?

[Jason Cooper]

Ok, 90% there...

On Fri, Oct 04, 2019 at 01:10:44AM +0000, Jason Cooper wrote:
> Hi all,
> 
> On Thu, Oct 03, 2019 at 02:40:16PM +0000, Jason Cooper wrote:
> > On Wed, Oct 02, 2019 at 01:33:22PM +0000, Gabor Nagy (gnagy) wrote:
> > > What Fabien suggested just rang a bell for me, but wouldn't
> > > "context-values" or "context-lookup" template functions  suit your
> > > need?
> > > 
> > > I have copied the admin guide's referring part:
> > > 
> > > context-values
> > > Syntax:
> > > $(context-values $name-value1 $name-value2 ...)
> > > Description: The context-values template function returns a list of
> > > every occurrence of the specified name-value pairs from the entire
> > > context. For example, if the context contains multiple messages, the
> > > $(context-values ${HOST}) template function will return a comma-
> > > separated list of the ${HOST} values that appear in the context.
> > 
> > Hot damn!  I think so!  It looks like I'll need `implode()` as well to
> > string together the messages from `context-values()` separated by
> > '\n'...
> 
> Well, I'm close :) Here's a dump to a log file destination for a single
> grouping-by() match:
> 

...

> The empty lines above are deliberately retained.  So, I get 8 empty lines,
> then my expected output, with the last line inexplicably doubled, and
> then one extra empty line.  Clearly I don't have this mastered yet.  :-)

This was due to placing the '\n' in the template, vice in the
construction of the multi-line message.  Basically the template was
being used for normal, un-grouping-by() lines.  So, MSGS macro was
empty, leading to the printed empty line.

I removed that '\n' from the template, and placed it outside the
'$(implode ...)'.  This now makes almost everything nice.

Except, I still get the last line of the (matching the 'trigger()') in my
group.  See below:

```
worker-dev[52081adeeae5e0b2] IAD:
    2019-10-04T15:14:21.766Z INFO Version: v0.12-4-gba793db5a79a
    2019-10-04T15:14:21.766Z INFO POST api-dev.example.com/v1/verifyReceipt called by: WWW.XXX.YYY.ZZZ
    2019-10-04T15:14:21.804Z DEBUG Receipt unchanged since last verified
    2019-10-04T15:14:22.535Z CRIT TypeError: Cannot read property 'duration_ms' of undefined
    2019-10-04T15:14:22.535Z CRIT     at updateProfileShopify (worker.js:698:71)
    2019-10-04T15:14:22.535Z CRIT     at async buildProfile (worker.js:583:15)
    2019-10-04T15:14:22.535Z CRIT     at async verifyReceipt (worker.js:1310:23)
    2019-10-04T15:14:22.535Z CRIT     at async failsafe (worker.js:65:24)
    2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error
    2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error
```

I suspect this might be a legit bug in '$(context-lookup ...)', but it's
really not the right tool for the job.  Rather, I should be using
'$(context-values ...)'

The reason I care about this is that 'program()' destination is a
long-running process that will have to split input into groups of lines
for sending discrete emails.  I can't use the 'smtp()' destination
because I have to log into my smtp server (I currently use sSMTP to do
this for other automated email tasks on this box).

The easiest way to detect the end of discrete group is with 'BAIL'.
Unfortunate, the extra BAIL line will screw that up. :-/

thx,

Jason.

[edits below for those curious]

> Here's the relevant portions of my syslog-ng config:
> 
> ```
> @version: 3.22
> 
> # common
> parser nginx-lua-parser {
> 	json-parser (prefix(".json."));
> };
> 
> parser alert-parser {
>    grouping-by(
>      key("${.json.rayid}")
>      having( "${.json.level}" == "CRIT" )
>      trigger(match("BAIL" value(".json.message")))
>      aggregate(
>        value("MSGS" "${.json.script}[${.json.rayid}] ${.json.colo}:\n$(implode '\n' $(context-lookup ('x' == 'x') $(implode ' ' '   ' ${.json.timestamp} ${.json.level} ${.json.message})))")

        value("MSGS" "${.json.script}[${.json.rayid}] ${.json.colo}:\n$(implode '\n' $(context-lookup ('x' == 'x') $(implode ' ' '   ' ${.json.timestamp} ${.json.level} ${.json.message})))\n")

Just added '\n' at the end.

>        inherit-mode("context")
>      )
>      inject-mode("pass-through")
>      timeout(10)
>    );
> };
> 
> template alert-template "${MSGS}\n";

 template alert-template "${MSGS}";

Just removed the '\n' here.

> 
> source worker-src {
> 	unix-stream("/var/run/nginx-lua/worker.sock", group(nginx) flags(no-parse));
> };
> 
> # development logs
> filter worker-dev-filter {match("-dev" value (".json.script"));};
> 
> destination worker-dev-alert-dest {
> 	file("/var/log/worker/alert-dev.log" template(alert-template));
> };
> 
> log {
> 	source(worker-src);
> 	parser(nginx-lua-parser);
> 	filter(worker-dev-filter);
> 	parser(alert-parser);
> 	destination(worker-dev-alert-dest);
> };
> ```