I'm trying to clean up the ad hoc logging in a collection of applications started on an embedded device using the daemon utility. Example: /usr/bin/daemon --name=myapp --output=user.debug --chdir=/MyApp -- /MyApp/bin/myapp There are 10 different applications started this way, written by different individuals/teams, and our syslog-ng.conf file currently looks something like this: @version:3.3.6 source s_local { system(); internal(); }; destination d_localfile { file("/var/log/messages" suppress(30)); }; log { source(s_local); destination(d_localfile); }; Because the --output=user.debug option to the daemon utility will cause the applications' stdout/stderr to be redirected to syslog, I'm a little worried about 'rogue' printf()'s filling up the logs with stuff like: Jan 1 00:00:54 (none) myapp: Battery voltage is 11.07 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.10 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.07 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.14 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.13 volts Jan 1 00:00:55 (none) myapp: Battery voltage is 11.09 volts Jan 1 00:00:55 (none) myapp: Battery voltage is 11.14 volts ... Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs: Jan 1 00:55:11 myapp: Battery voltage is 11.07 volts Jan 1 00:55:14 myapp: Last message 'Battery voltage is 1' repeated with nearly identical content 418 times, suppressed by syslog-ng on flahblargle A few of my colleagues have suggested that this is probably more trouble than it's worth, and we should just fix the spammy app(s) rather than try to 'outsmart' them. And I think I agree with them, but... it was my idea to move to syslog-ng, so there'll be a bit more egg on my face than theirs if we accidentally allow a rogue printf() in a hot loop somewhere to escape into production. `:-} Any advice/insight much appreciated, thanks!
I don't believe that syslog-ng can do that, but I use SEC (simple event correlator) for that purpose. On Thu, May 22, 2014 at 10:55 AM, Evade Flow <evadeflow@gmail.com> wrote:
I'm trying to clean up the ad hoc logging in a collection of applications started on an embedded device using the daemon utility. Example:
/usr/bin/daemon --name=myapp --output=user.debug --chdir=/MyApp -- /MyApp/bin/myapp
There are 10 different applications started this way, written by different individuals/teams, and our syslog-ng.conf file currently looks something like this:
@version:3.3.6 source s_local { system(); internal(); }; destination d_localfile { file("/var/log/messages" suppress(30)); }; log { source(s_local); destination(d_localfile); };
Because the --output=user.debug option to the daemon utility will cause the applications' stdout/stderr to be redirected to syslog, I'm a little worried about 'rogue' printf()'s filling up the logs with stuff like:
Jan 1 00:00:54 (none) myapp: Battery voltage is 11.07 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.10 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.07 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.14 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.13 volts Jan 1 00:00:55 (none) myapp: Battery voltage is 11.09 volts Jan 1 00:00:55 (none) myapp: Battery voltage is 11.14 volts ...
Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs:
Jan 1 00:55:11 myapp: Battery voltage is 11.07 volts Jan 1 00:55:14 myapp: Last message 'Battery voltage is 1' repeated with nearly identical content 418 times, suppressed by syslog-ng on flahblargle
A few of my colleagues have suggested that this is probably more trouble than it's worth, and we should just fix the spammy app(s) rather than try to 'outsmart' them. And I think I agree with them, but... it was my idea to move to syslog-ng, so there'll be a bit more egg on my face than theirs if we accidentally allow a rogue printf() in a hot loop somewhere to escape into production. `:-}
Any advice/insight much appreciated, thanks!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs:
Here's the way I'd try doing it, based on your example: 1) Use filters to separate the "myapp: Battery voltage is N volts" message flow from all other messages. 2) Tag the "Regular flow" of messages using e.g. the tag "keep" Optionally tag the other stream using "drop" 3) Parse the "myapp" messages using a patterndb parser, correlating all messages from the same host, then trigger an action upon timeout or reaching threshold (use context-length macro). The generated message can contain useful information from the context, e.g. number of similar messages (CONTEXT_LENGTH), average of voltages (would probably need some template function acrobatics), etc. 4) Tag the action generated message using "keep" 5) Filter out all messages not having the tag "keep". This way the individual "myapp" messages won't be logged. Only the "regular flow" of messages and the triggered messages will.
Thanks for the response, it really helped me see how syslog-ng can break apart an input stream and re-assemble it. Seems very powerful! With regard to filtering/pattern matching: do I need to know the patterns to be matched in advance? In my case, unfortunately, I do not. Some of these apps were written by people I've never met, and the SLOC count for all apps taken together is well over 1 million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those? On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs:
Here's the way I'd try doing it, based on your example:
1) Use filters to separate the "myapp: Battery voltage is N volts" message flow from all other messages. 2) Tag the "Regular flow" of messages using e.g. the tag "keep" Optionally tag the other stream using "drop" 3) Parse the "myapp" messages using a patterndb parser, correlating all messages from the same host, then trigger an action upon timeout or reaching threshold (use context-length macro). The generated message can contain useful information from the context, e.g. number of similar messages (CONTEXT_LENGTH), average of voltages (would probably need some template function acrobatics), etc. 4) Tag the action generated message using "keep" 5) Filter out all messages not having the tag "keep". This way the individual "myapp" messages won't be logged. Only the "regular flow" of messages and the triggered messages will.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You could also try LogZilla - it automatically deduplicates messages. ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Thu, May 22, 2014 at 4:54 PM, Evade Flow <evadeflow@gmail.com> wrote:
Thanks for the response, it really helped me see how syslog-ng can break apart an input stream and re-assemble it. Seems very powerful!
With regard to filtering/pattern matching: do I need to know the patterns to be matched in advance? In my case, unfortunately, I do not. Some of these apps were written by people I've never met, and the SLOC count for all apps taken together is well over 1 million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those?
On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs:
Here's the way I'd try doing it, based on your example:
1) Use filters to separate the "myapp: Battery voltage is N volts" message flow from all other messages. 2) Tag the "Regular flow" of messages using e.g. the tag "keep" Optionally tag the other stream using "drop" 3) Parse the "myapp" messages using a patterndb parser, correlating all messages from the same host, then trigger an action upon timeout or reaching threshold (use context-length macro). The generated message can contain useful information from the context, e.g. number of similar messages (CONTEXT_LENGTH), average of voltages (would probably need some template function acrobatics), etc. 4) Tag the action generated message using "keep" 5) Filter out all messages not having the tag "keep". This way the individual "myapp" messages won't be logged. Only the "regular flow" of messages and the triggered messages will.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
you may want to take a look at this presentation http://www.occam.com/sa/CentralizedLogging2012.pdf SEC deduping does not require you to know the patterns ahead of time. But YMMV ;) On Thu, May 22, 2014 at 3:38 PM, Clayton Dukes <cdukes@gmail.com> wrote:
You could also try LogZilla - it automatically deduplicates messages.
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Thu, May 22, 2014 at 4:54 PM, Evade Flow <evadeflow@gmail.com> wrote:
Thanks for the response, it really helped me see how syslog-ng can break apart an input stream and re-assemble it. Seems very powerful!
With regard to filtering/pattern matching: do I need to know the patterns to be matched in advance? In my case, unfortunately, I do not. Some of these apps were written by people I've never met, and the SLOC count for all apps taken together is well over 1 million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those?
On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs:
Here's the way I'd try doing it, based on your example:
1) Use filters to separate the "myapp: Battery voltage is N volts" message flow from all other messages. 2) Tag the "Regular flow" of messages using e.g. the tag "keep" Optionally tag the other stream using "drop" 3) Parse the "myapp" messages using a patterndb parser, correlating all messages from the same host, then trigger an action upon timeout or reaching threshold (use context-length macro). The generated message can contain useful information from the context, e.g. number of similar messages (CONTEXT_LENGTH), average of voltages (would probably need some template function acrobatics), etc. 4) Tag the action generated message using "keep" 5) Filter out all messages not having the tag "keep". This way the individual "myapp" messages won't be logged. Only the "regular flow" of messages and the triggered messages will.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks for mentioning SEC, and the link to the (quite good) presentation. We've decided that pre-processing the logs is too much work, and probably a bad idea, so we're just going to attempt to ensure that none of the apps are too chatty, and rely on our log rotation scheme to deal with the fallout from any corner cases that cause the logs to 'blow up'. But SEC looks like a slam dunk for post-processing/analyzing logs collected from devices in the field. On Thu, May 22, 2014 at 7:00 PM, Orangepeel Beef <orangepeelbeef@gmail.com>wrote:
you may want to take a look at this presentation http://www.occam.com/sa/CentralizedLogging2012.pdf SEC deduping does not require you to know the patterns ahead of time. But YMMV ;)
On Thu, May 22, 2014 at 3:38 PM, Clayton Dukes <cdukes@gmail.com> wrote:
You could also try LogZilla - it automatically deduplicates messages.
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Thu, May 22, 2014 at 4:54 PM, Evade Flow <evadeflow@gmail.com> wrote:
Thanks for the response, it really helped me see how syslog-ng can break apart an input stream and re-assemble it. Seems very powerful!
With regard to filtering/pattern matching: do I need to know the patterns to be matched in advance? In my case, unfortunately, I do not. Some of these apps were written by people I've never met, and the SLOC count for all apps taken together is well over 1 million lines of code. I suppose I could run the app hundreds of times under various loads, collect representative log files, and build the pattern database from those?
On Thu, May 22, 2014 at 2:58 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Thu, May 22, 2014 at 01:55:58PM -0400, Evade Flow wrote:
Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs:
Here's the way I'd try doing it, based on your example:
1) Use filters to separate the "myapp: Battery voltage is N volts" message flow from all other messages. 2) Tag the "Regular flow" of messages using e.g. the tag "keep" Optionally tag the other stream using "drop" 3) Parse the "myapp" messages using a patterndb parser, correlating all messages from the same host, then trigger an action upon timeout or reaching threshold (use context-length macro). The generated message can contain useful information from the context, e.g. number of similar messages (CONTEXT_LENGTH), average of voltages (would probably need some template function acrobatics), etc. 4) Tag the action generated message using "keep" 5) Filter out all messages not having the tag "keep". This way the individual "myapp" messages won't be logged. Only the "regular flow" of messages and the triggered messages will.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Clayton Dukes
-
Evade Flow
-
Fabien Wernli
-
Orangepeel Beef