All, I have Cloudflare Workers sending log messages to my syslog-ng server. There is an intermediary step where nginx receives a POST with a JSON body containing all the messages for a given run (try{}catch(e){} ensures it sends the accumulated messages at the end of each run). Some Lua in the nginx server parses the JSON and separates the array of messages into individual writes to the unix domain socket syslog-ng is listening on. This whole setup is tested, and working great. I even get the stack trace (JS, not a core dump) when I screw up. :-) Now, I'd like to auto-fire an email on certain events. e.g. when .json.level == "CRIT" (script threw an error). The trick is, I'd like to dump *all* the log messages for the matching run (only a single line from a run might match) to the email / process destination. See the commented out alert_parser{}, below. The last message of every run (equivalent to a unique .json.rayid, which I treat as a process id) always starts with 'BAIL'. So I use that as a trigger. I've been using $searchengine[*] and not been able to figure out how to send /all/ of the messages in a group to aggregate(). I even dug into the source a bit and saw that I can reference messages by '@2' for the second message back. But I see no way to a) get the number of messages, b) loop through the messages, or c) reference all of them, e.g. '@*'. Has anyone solved this problem? tia, Jason. ----------->8--------------------------------------------------------- @version: 3.22 # common parser nginx-lua-parser { json-parser (prefix(".json.")); }; #parser alert_parser { # grouping-by( # key("${json.rayid}") # scope("process") # timeout(5) # having("CRIT") # trigger("BAIL") # aggregate( # value("MESSAGE" "\n\n") # inherit-mode("context") # ) # inject-mode("pass-through") # ); #}; # template nginx-lua-template "${.json.timestamp} ${.json.colo} ${.json.script}[${.json.rayid}]: ${.json.level} ${.json.message}\n"; source worker-src { unix-stream("/var/run/nginx-lua/worker.sock", group(nginx) flags(no-parse)); }; # development logs filter worker-dev-filter {match("-dev" value (".json.script"));}; destination worker-dev-dest { file("/var/log/worker/development.log" template(nginx-lua-template)); }; log { source(worker-src); parser(nginx-lua-parser); filter(worker-dev-filter); destination(worker-dev-dest); }; # production logs filter worker-prod-filter {match("-prod" value (".json.script"));}; destination worker-prod-dest { file("/var/log/worker/production.log" template(nginx-lua-template)); }; log { source(worker-src); parser(nginx-lua-parser); filter(worker-prod-filter); destination(worker-prod-dest); };
Hi Jason, You can use the `$(context-length)` function to get the first message, but that won't help you as you can't (AFAIK) loop through the backref index. What you might be able to do is use the `$(grep)` function maybe it's possible to make it return all matches ? If grouping-by() can't do it, patterndb does: it allows for using the `value` parameter (as in `aggregate()`) for every matching message in the context, not only in the last one. Thus you can concatenate the values of a macro, e.g. in order to collect all `HOST` macros in a given context. I'd open a github issue in order to allow for using `value` in every message matching the context, which sounds like a very useful addition to me. cheers
Hi again, On Wed, Oct 02, 2019 at 09:40:21AM +0200, Fabien Wernli wrote:
message, but that won't help you as you can't (AFAIK) loop through the backref index. What you might be able to do is use the `$(grep)` function
Actually, you cannot yet, but might be able to soon(ish): https://github.com/syslog-ng/syslog-ng/issues/2926#issuecomment-536985104
What Fabien suggested just rang a bell for me, but wouldn't "context-values" or "context-lookup" template functions suit your need? I have copied the admin guide's referring part: context-values Syntax: $(context-values $name-value1 $name-value2 ...) Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma- separated list of the ${HOST} values that appear in the context. Sorry if I'm wrong, these are just my 2 cents. Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli <wernli@in2p3.fr> Sent: Wednesday, October 2, 2019 14:49 To: Fabien Wernli <wernli@in2p3.fr> Cc: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] group-by() send all messages to destination() ? CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi again, On Wed, Oct 02, 2019 at 09:40:21AM +0200, Fabien Wernli wrote:
message, but that won't help you as you can't (AFAIK) loop through the backref index. What you might be able to do is use the `$(grep)` function
Actually, you cannot yet, but might be able to soon(ish): https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fissues%2F2926%23issuecomment-536985104&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7Cd8fd2a28102c40e4b3e108d7473701e0%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637056173904330670&sdata=bL%2F1iPWAL%2BIlisuFxI7narwNHX%2BhfN%2F4XW5XxVC94kY%3D&reserved=0 ______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7Cd8fd2a28102c40e4b3e108d7473701e0%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637056173904330670&sdata=NG8ADuvQSRrE2KvMd3GqNo3GmeVB6dfdLcZx7kLyhHM%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7Cd8fd2a28102c40e4b3e108d7473701e0%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637056173904330670&sdata=Q5Z6vt3LfDPQUxAoWZAGZuApFeAq4djyoIe6jGYsRPk%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7Cd8fd2a28102c40e4b3e108d7473701e0%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637056173904330670&sdata=EIOR2ql5puj1UnWHgaRWqvm0p066Xrp0cAqY8nTEp1I%3D&reserved=0
Hi Gabor, On Wed, Oct 02, 2019 at 01:33:22PM +0000, Gabor Nagy (gnagy) wrote:
What Fabien suggested just rang a bell for me, but wouldn't "context-values" or "context-lookup" template functions suit your need?
I have copied the admin guide's referring part:
context-values Syntax:
These look great, but I can't find them in the doc. Are they PE-only? If they're not, my search-fu must be broken. In any case, these functions should be referenced here: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
Hi Fabien, I've found them at the template functions chapter: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... I agree with you, that these should be referenced and explained in grouping-by/patterndb chapters. ________________________________ From: Fabien Wernli <wernli@in2p3.fr> Sent: Wednesday, October 2, 2019 16:41 To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Cc: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: Re: [syslog-ng] group-by() send all messages to destination() ? CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
I've added an entry about that to our todo list. Cheers, Robert ________________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Sent: Wednesday, October 2, 2019 17:05 To: wernli@in2p3.fr Cc: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] group-by() send all messages to destination() ? CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi Fabien, I've found them at the template functions chapter: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.23/administration-guide/63#TOPIC-1268645<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.23%2Fadministration-guide%2F63%23TOPIC-1268645&data=02%7C01%7Crobert.fekete%40oneidentity.com%7C0560e9ad9dcc4b7b8ab208d74749fdaa%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637056255424811608&sdata=R%2BaB3889hrLedXkR1b6Wy0lNJqK2ph9X2lQHDEu%2B5Ms%3D&reserved=0> I agree with you, that these should be referenced and explained in grouping-by/patterndb chapters. ________________________________ From: Fabien Wernli <wernli@in2p3.fr> Sent: Wednesday, October 2, 2019 16:41 To: Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Cc: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: Re: [syslog-ng] group-by() send all messages to destination() ? CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi Gabor! On Wed, Oct 02, 2019 at 01:33:22PM +0000, Gabor Nagy (gnagy) wrote:
What Fabien suggested just rang a bell for me, but wouldn't "context-values" or "context-lookup" template functions suit your need?
I have copied the admin guide's referring part:
context-values Syntax: $(context-values $name-value1 $name-value2 ...) Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma- separated list of the ${HOST} values that appear in the context.
Hot damn! I think so! It looks like I'll need `implode()` as well to string together the messages from `context-values()` separated by '\n'... thx, Jason.
Hi all, On Thu, Oct 03, 2019 at 02:40:16PM +0000, Jason Cooper wrote:
On Wed, Oct 02, 2019 at 01:33:22PM +0000, Gabor Nagy (gnagy) wrote:
What Fabien suggested just rang a bell for me, but wouldn't "context-values" or "context-lookup" template functions suit your need?
I have copied the admin guide's referring part:
context-values Syntax: $(context-values $name-value1 $name-value2 ...) Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma- separated list of the ${HOST} values that appear in the context.
Hot damn! I think so! It looks like I'll need `implode()` as well to string together the messages from `context-values()` separated by '\n'...
Well, I'm close :) Here's a dump to a log file destination for a single grouping-by() match: ``` worker-dev[52030324bc41cf34] IAD: 2019-10-04T00:24:20.864Z INFO Version: v0.12-4-g4ff99a0cb938 2019-10-04T00:24:20.864Z INFO POST api-dev.example.com/v1/verifyReceipt called by: WWW.XXX.YYY.ZZZ 2019-10-04T00:24:21.284Z DEBUG Receipt unchanged since last verified 2019-10-04T00:24:26.227Z CRIT TypeError: Cannot read property 'duration_ms' of undefined 2019-10-04T00:24:26.227Z CRIT at updateProfileShopify (worker.js:698:71) 2019-10-04T00:24:26.227Z CRIT at async buildProfile (worker.js:583:15) 2019-10-04T00:24:26.227Z CRIT at async verifyReceipt (worker.js:1310:23) 2019-10-04T00:24:26.227Z CRIT at async failsafe (worker.js:65:24) 2019-10-04T00:24:26.227Z INFO BAIL(500) ERROR: Internal server error 2019-10-04T00:24:26.227Z INFO BAIL(500) ERROR: Internal server error ``` The empty lines above are deliberately retained. So, I get 8 empty lines, then my expected output, with the last line inexplicably doubled, and then one extra empty line. Clearly I don't have this mastered yet. :-) A possible coincidence is that there are 8 non-duplicated messages for 8 empty lines. Here's the relevant portions of my syslog-ng config: ``` @version: 3.22 # common parser nginx-lua-parser { json-parser (prefix(".json.")); }; parser alert-parser { grouping-by( key("${.json.rayid}") having( "${.json.level}" == "CRIT" ) trigger(match("BAIL" value(".json.message"))) aggregate( value("MSGS" "${.json.script}[${.json.rayid}] ${.json.colo}:\n$(implode '\n' $(context-lookup ('x' == 'x') $(implode ' ' ' ' ${.json.timestamp} ${.json.level} ${.json.message})))") inherit-mode("context") ) inject-mode("pass-through") timeout(10) ); }; template alert-template "${MSGS}\n"; source worker-src { unix-stream("/var/run/nginx-lua/worker.sock", group(nginx) flags(no-parse)); }; # development logs filter worker-dev-filter {match("-dev" value (".json.script"));}; destination worker-dev-alert-dest { file("/var/log/worker/alert-dev.log" template(alert-template)); }; log { source(worker-src); parser(nginx-lua-parser); filter(worker-dev-filter); parser(alert-parser); destination(worker-dev-alert-dest); }; ``` So, why I am getting empty lines? and why is the trigger line duplicated? Thanks! Jason.
Ok, 90% there... On Fri, Oct 04, 2019 at 01:10:44AM +0000, Jason Cooper wrote:
Hi all,
On Thu, Oct 03, 2019 at 02:40:16PM +0000, Jason Cooper wrote:
On Wed, Oct 02, 2019 at 01:33:22PM +0000, Gabor Nagy (gnagy) wrote:
What Fabien suggested just rang a bell for me, but wouldn't "context-values" or "context-lookup" template functions suit your need?
I have copied the admin guide's referring part:
context-values Syntax: $(context-values $name-value1 $name-value2 ...) Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma- separated list of the ${HOST} values that appear in the context.
Hot damn! I think so! It looks like I'll need `implode()` as well to string together the messages from `context-values()` separated by '\n'...
Well, I'm close :) Here's a dump to a log file destination for a single grouping-by() match:
...
The empty lines above are deliberately retained. So, I get 8 empty lines, then my expected output, with the last line inexplicably doubled, and then one extra empty line. Clearly I don't have this mastered yet. :-)
This was due to placing the '\n' in the template, vice in the construction of the multi-line message. Basically the template was being used for normal, un-grouping-by() lines. So, MSGS macro was empty, leading to the printed empty line. I removed that '\n' from the template, and placed it outside the '$(implode ...)'. This now makes almost everything nice. Except, I still get the last line of the (matching the 'trigger()') in my group. See below: ``` worker-dev[52081adeeae5e0b2] IAD: 2019-10-04T15:14:21.766Z INFO Version: v0.12-4-gba793db5a79a 2019-10-04T15:14:21.766Z INFO POST api-dev.example.com/v1/verifyReceipt called by: WWW.XXX.YYY.ZZZ 2019-10-04T15:14:21.804Z DEBUG Receipt unchanged since last verified 2019-10-04T15:14:22.535Z CRIT TypeError: Cannot read property 'duration_ms' of undefined 2019-10-04T15:14:22.535Z CRIT at updateProfileShopify (worker.js:698:71) 2019-10-04T15:14:22.535Z CRIT at async buildProfile (worker.js:583:15) 2019-10-04T15:14:22.535Z CRIT at async verifyReceipt (worker.js:1310:23) 2019-10-04T15:14:22.535Z CRIT at async failsafe (worker.js:65:24) 2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error 2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error ``` I suspect this might be a legit bug in '$(context-lookup ...)', but it's really not the right tool for the job. Rather, I should be using '$(context-values ...)' The reason I care about this is that 'program()' destination is a long-running process that will have to split input into groups of lines for sending discrete emails. I can't use the 'smtp()' destination because I have to log into my smtp server (I currently use sSMTP to do this for other automated email tasks on this box). The easiest way to detect the end of discrete group is with 'BAIL'. Unfortunate, the extra BAIL line will screw that up. :-/ thx, Jason. [edits below for those curious]
Here's the relevant portions of my syslog-ng config:
``` @version: 3.22
# common parser nginx-lua-parser { json-parser (prefix(".json.")); };
parser alert-parser { grouping-by( key("${.json.rayid}") having( "${.json.level}" == "CRIT" ) trigger(match("BAIL" value(".json.message"))) aggregate( value("MSGS" "${.json.script}[${.json.rayid}] ${.json.colo}:\n$(implode '\n' $(context-lookup ('x' == 'x') $(implode ' ' ' ' ${.json.timestamp} ${.json.level} ${.json.message})))")
value("MSGS" "${.json.script}[${.json.rayid}] ${.json.colo}:\n$(implode '\n' $(context-lookup ('x' == 'x') $(implode ' ' ' ' ${.json.timestamp} ${.json.level} ${.json.message})))\n") Just added '\n' at the end.
inherit-mode("context") ) inject-mode("pass-through") timeout(10) ); };
template alert-template "${MSGS}\n";
template alert-template "${MSGS}"; Just removed the '\n' here.
source worker-src { unix-stream("/var/run/nginx-lua/worker.sock", group(nginx) flags(no-parse)); };
# development logs filter worker-dev-filter {match("-dev" value (".json.script"));};
destination worker-dev-alert-dest { file("/var/log/worker/alert-dev.log" template(alert-template)); };
log { source(worker-src); parser(nginx-lua-parser); filter(worker-dev-filter); parser(alert-parser); destination(worker-dev-alert-dest); }; ```
mumbling to self... :-) [TL;DR: see below if you are having trouble with `having()` not working properly] Ulitmately, it appears that grouping-by() was designed around the concept of catching a group with a consistent number of messages. Most of my frustration is centered around a) not nowing how many lines the stacktrace is, and b) not knowing where the exception will occur, so I also don't know which execution flow, and thus how many lines, will have an exception. So, I doubly-don't know how many lines will be in the resulting grouping-by context. :-( On Fri, Oct 04, 2019 at 04:48:09PM +0000, Jason Cooper wrote: ...
Except, I still get the last line of the (matching the 'trigger()') in my group. See below:
``` worker-dev[52081adeeae5e0b2] IAD: 2019-10-04T15:14:21.766Z INFO Version: v0.12-4-gba793db5a79a 2019-10-04T15:14:21.766Z INFO POST api-dev.example.com/v1/verifyReceipt called by: WWW.XXX.YYY.ZZZ 2019-10-04T15:14:21.804Z DEBUG Receipt unchanged since last verified 2019-10-04T15:14:22.535Z CRIT TypeError: Cannot read property 'duration_ms' of undefined 2019-10-04T15:14:22.535Z CRIT at updateProfileShopify (worker.js:698:71) 2019-10-04T15:14:22.535Z CRIT at async buildProfile (worker.js:583:15) 2019-10-04T15:14:22.535Z CRIT at async verifyReceipt (worker.js:1310:23) 2019-10-04T15:14:22.535Z CRIT at async failsafe (worker.js:65:24) 2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error 2019-10-04T15:14:22.535Z INFO BAIL(500) ERROR: Internal server error ```
I suspect this might be a legit bug in '$(context-lookup ...)', but it's really not the right tool for the job. Rather, I should be using '$(context-values ...)'
$(context-values ...) works great, I no longer need the silly always-true condition.
The reason I care about this is that 'program()' destination is a long-running process that will have to split input into groups of lines for sending discrete emails. I can't use the 'smtp()' destination because I have to log into my smtp server (I currently use sSMTP to do this for other automated email tasks on this box).
The easiest way to detect the end of discrete group is with 'BAIL'. Unfortunate, the extra BAIL line will screw that up. :-/
meh. The bug is still present when using $(context-values ...). However, my summary line is the only line not starting with leading whitespace, so I can just trigger off of that instead. Then there won't be anything to back out should a) I figure out what I did to duplicate the last message, or b) the bug gets fixed.
``` @version: 3.22
# common parser nginx-lua-parser { json-parser (prefix(".json.")); };
parser alert-parser { grouping-by( key("${.json.rayid}")
having( "${.json.level}" == "CRIT" )
Yeah... This is some arcane shit. After beating my head against the wall for days, this ended up being the problem. Depending on how I tweaked this, either nothing would match (no aggregate message ever generated), or everything would match. :-/ Turns out, this sentence is critical: "Note that the having() filter has access to every message of the context." The *only* real world example of using grouping-by() is here: https://balagetech.com/monitor-network-traffic-openwrt-syslog-ng/ which contains this line, but also doesn't draw attention to it: having( "${UNIXTIME}@2" ne "1" ) The trick here is that *no one* mentions that you _must_ reference a specific message within the context. There doesn't seem to be a way to say "If you see /any/ messages in the context with, e.g. LEVEL == "CRIT", fire off the aggregate. :-/ Luckily, since my exception handler fires off the final message as "Internal server error", the following is now working beautifully: having( "${.json.message}@1" == "ERROR: Internal server error") Hope this saves someone some frustration down the road... thx, Jason.
participants (4)
-
Fabien Wernli
-
Gabor Nagy (gnagy)
-
Jason Cooper
-
Robert Fekete (rfekete)