Hi,

I was involved in a number of projects where syslog-ng was used in a relatively complex setup: we wanted to structure incoming log messages and transform them into nice & clean JSON messages.

In a number of cases, this required several parsers to get right (csv-parser, db-parser and sometimes regexps) and the resulting configuration was pretty complicated.

I've figured, that if we have a domain specific language in the form of the syslog-ng configuration file, we should also have a debugger that makes it easy to find out what went wrong. And this is what I have did, some proof-of-concept code is available on the f/debugger branch on github:

https://github.com/balabit/syslog-ng/tree/f/debugger

Here's how it works:

# start up syslog-ng with the -i option to indicate we
# want the debugger enabled
$ sbin/syslog-ng -iFe

Once syslog-ng starts up, you'll see something like this:
Waiting for breakpoint...

For now, all processing elements within syslog-ng (called LogPipe) breaks into the debugger. Later, I want to implement real breakpoints, where you can selectively attach breakpoints to elements in the configuration file.

# on another console send a log message to syslog-ng:
$ logger almafa

This is what happens on the syslog-ng side:

Breakpoint hit etc/syslog-ng.conf:11:2
11           unix-stream("log");
Dec 15 16:14:05  bazsi: almafa
(syslog-ng)

At a breakpoint, syslog-ng displays where the break has occurred, quotes the source line and displays the message that is being processed.

Now it becomes interactive, you can inspect the message with various commands:

(syslog-ng) help
syslog-ng interactive console, the following commands are available

  help, h, or ?            Display this help
  continue or c            Continue until the next breakpoint
  print, p                 Print the current log message
  drop, d                  Drop the current message
  quit, q                  Tell syslog-ng to exit
(syslog-ng) p $MSG
almafa
(syslog-ng) p
MESSAGE=almafa
PROGRAM=bazsi
LEGACY_MSGHDR=bazsi:
.unix.pid=18703
.unix.uid=1000
.unix.gid=1000
TAGS=

(syslog-ng) p "$(format-json --key *)"
{"_unix":{"uid":"1000","pid":"18703","gid":"1000"},  ... }


With the "continue" command, you can request syslog-ng to proceed to the next LogPipe, where it stops again. "drop" drops the current message, "quit" tells syslog-ng to exit.

I would really like to extend this to further ways, like:
  • real breakpoints, instead of stopping everywhere
  • inject messages
  • change messages to make configuration testable
  • single-step (to the next primitive LogPipe) and step-over (to the next 'real' LogPipe that is found in the configuration file)
  • interactive REPL to template functions to make it easier to interact with them

What do you think?

--
Bazsi