mumbling to self... :-) [TL;DR: see below if you are having trouble with `having()` not working properly] Ulitmately, it appears that grouping-by() was designed around the concept of catching a group with a consistent number of messages. Most of my frustration is centered around a) not nowing how many lines the stacktrace is, and b) not knowing where the exception will occur, so I also don't know which execution flow, and thus how many lines, will have an exception. So, I doubly-don't know how many lines will be in the resulting grouping-by context. :-( On Fri, Oct 04, 2019 at 04:48:09PM +0000, Jason Cooper wrote: ...
Except, I still get the last line of the (matching the 'trigger()') in my group. See below:
``` worker-dev[52081adeeae5e0b2] IAD: 2019-10-04T15:14:21.766Z INFO Version: v0.12-4-gba793db5a79a 2019-10-04T15:14:21.766Z INFO POST api-dev.example.com/v1/verifyReceipt called by: WWW.XXX.YYY.ZZZ 2019-10-04T15:14:21.804Z DEBUG Receipt unchanged since last verified 2019-10-04T15:14:22.535Z CRIT TypeError: Cannot read property 'duration_ms' of undefined 2019-10-04T15:14:22.535Z CRIT at updateProfileShopify (worker.js:698:71) 2019-10-04T15:14:22.535Z CRIT at async buildProfile (worker.js:583:15) 2019-10-04T15:14:22.535Z CRIT at async verifyReceipt (worker.js:1310:23) 2019-10-04T15:14:22.535Z CRIT at async failsafe (worker.js:65:24) 2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error 2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error ```
I suspect this might be a legit bug in '$(context-lookup ...)', but it's really not the right tool for the job. Rather, I should be using '$(context-values ...)'
$(context-values ...) works great, I no longer need the silly always-true condition.
The reason I care about this is that 'program()' destination is a long-running process that will have to split input into groups of lines for sending discrete emails. I can't use the 'smtp()' destination because I have to log into my smtp server (I currently use sSMTP to do this for other automated email tasks on this box).
The easiest way to detect the end of discrete group is with 'BAIL'. Unfortunate, the extra BAIL line will screw that up. :-/
meh. The bug is still present when using $(context-values ...). However, my summary line is the only line not starting with leading whitespace, so I can just trigger off of that instead. Then there won't be anything to back out should a) I figure out what I did to duplicate the last message, or b) the bug gets fixed.
``` @version: 3.22
# common parser nginx-lua-parser { json-parser (prefix(".json.")); };
parser alert-parser { grouping-by( key("${.json.rayid}")
having( "${.json.level}" == "CRIT" )
Yeah... This is some arcane shit. After beating my head against the wall for days, this ended up being the problem. Depending on how I tweaked this, either nothing would match (no aggregate message ever generated), or everything would match. :-/ Turns out, this sentence is critical: "Note that the having() filter has access to every message of the context." The *only* real world example of using grouping-by() is here: https://balagetech.com/monitor-network-traffic-openwrt-syslog-ng/ which contains this line, but also doesn't draw attention to it: having( "${UNIXTIME}@2" ne "1" ) The trick here is that *no one* mentions that you _must_ reference a specific message within the context. There doesn't seem to be a way to say "If you see /any/ messages in the context with, e.g. LEVEL == "CRIT", fire off the aggregate. :-/ Luckily, since my exception handler fires off the final message as "Internal server error", the following is now working beautifully: having( "${.json.message}@1" == "ERROR: Internal server error") Hope this saves someone some frustration down the road... thx, Jason.