[syslog-ng] group-by() send all messages to destination() ?

[Jason Cooper]

mumbling to self... :-)

[TL;DR: see below if you are having trouble with `having()` not working
properly]

Ulitmately, it appears that grouping-by() was designed around the
concept of catching a group with a consistent number of messages.  Most
of my frustration is centered around a) not nowing how many lines the
stacktrace is, and b) not knowing where the exception will occur, so I
also don't know which execution flow, and thus how many lines, will have
an exception.  So, I doubly-don't know how many lines will be in the
resulting grouping-by context. :-(

On Fri, Oct 04, 2019 at 04:48:09PM +0000, Jason Cooper wrote:
...
> Except, I still get the last line of the (matching the 'trigger()') in my
> group.  See below:
> 
> ```
> worker-dev[52081adeeae5e0b2] IAD:
>     2019-10-04T15:14:21.766Z INFO Version: v0.12-4-gba793db5a79a
>     2019-10-04T15:14:21.766Z INFO POST api-dev.example.com/v1/verifyReceipt called by: WWW.XXX.YYY.ZZZ
>     2019-10-04T15:14:21.804Z DEBUG Receipt unchanged since last verified
>     2019-10-04T15:14:22.535Z CRIT TypeError: Cannot read property 'duration_ms' of undefined
>     2019-10-04T15:14:22.535Z CRIT     at updateProfileShopify (worker.js:698:71)
>     2019-10-04T15:14:22.535Z CRIT     at async buildProfile (worker.js:583:15)
>     2019-10-04T15:14:22.535Z CRIT     at async verifyReceipt (worker.js:1310:23)
>     2019-10-04T15:14:22.535Z CRIT     at async failsafe (worker.js:65:24)
>     2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error
>     2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error
> ```
> 
> I suspect this might be a legit bug in '$(context-lookup ...)', but it's
> really not the right tool for the job.  Rather, I should be using
> '$(context-values ...)'

$(context-values ...) works great, I no longer need the silly always-true
condition.

> The reason I care about this is that 'program()' destination is a
> long-running process that will have to split input into groups of lines
> for sending discrete emails.  I can't use the 'smtp()' destination
> because I have to log into my smtp server (I currently use sSMTP to do
> this for other automated email tasks on this box).
> 
> The easiest way to detect the end of discrete group is with 'BAIL'.
> Unfortunate, the extra BAIL line will screw that up. :-/

meh.  The bug is still present when using $(context-values ...).
However, my summary line is the only line not starting with leading
whitespace, so I can just trigger off of that instead.  Then there won't
be anything to back out should a) I figure out what I did to duplicate
the last message, or b) the bug gets fixed.

> > ```
> > @version: 3.22
> > 
> > # common
> > parser nginx-lua-parser {
> > 	json-parser (prefix(".json."));
> > };
> > 
> > parser alert-parser {
> >    grouping-by(
> >      key("${.json.rayid}")


> >      having( "${.json.level}" == "CRIT" )

Yeah... This is some arcane shit.  After beating my head against the
wall for days, this ended up being the problem.  Depending on how I
tweaked this, either nothing would match (no aggregate message ever
generated), or everything would match. :-/  Turns out, this sentence is
critical:

"Note that the having() filter has access to every message of the
context."

The *only* real world example of using grouping-by() is here:

  https://balagetech.com/monitor-network-traffic-openwrt-syslog-ng/

which contains this line, but also doesn't draw attention to it:

  having( "${UNIXTIME}@2" ne "1" )

The trick here is that *no one* mentions that you _must_ reference a
specific message within the context.  There doesn't seem to be a way to
say "If you see /any/ messages in the context with, e.g. LEVEL ==
"CRIT", fire off the aggregate. :-/

Luckily, since my exception handler fires off the final message as
"Internal server error", the following is now working beautifully:

  having( "${.json.message}@1" == "ERROR: Internal server error")

Hope this saves someone some frustration down the road...


thx,

Jason.