Hi,

I've Cced the syslog-ng list, so others can see this description as well.

The goal here is to change syslog-ng to be able to process input from stdin and produce output to stdout and exit when the EOF is reached, while processing a "potentially" simplified syslog-ng configuration block. A bit similar to how awk works, but instead of the awk language, use syslog-ng's constructs.

E.g.

$ syslog-ng -P 'parser { kv-parser(); }; destination { stdout(template("$(format-json *"))); };' < log-file-with-keywords.log > log-file-in-json

Where -P would be equivalent to --pipe, the next argument is a configuration snippet. This snippet would get all logs as read from stdin() and that would be default functionality provided by the --pipe construct. Probably --pipe would use a configuration file something like:

log {

source { stdin(); };

# program supplied by --pipe would be included here

pipe-program();
};

This is the basic idea, some customization would be useful, like:

being able to specify the source as well (so the above wouldn't be hardwired, only used as a default)
maybe use a default for output as well, so I wouldn't have to include it in the actual language
examples: a few examples how to process existing log files from postfix, iptables, snort/suricata, etc to convert their logs into json. This would be both a set of examples and could also drive the functionality further, so we don't miss important options.

Code wise, this would need something like:

stdin source: would probably only be an SCL wrapper around the existing file() driver; e.g. file("/dev/stdin"); check for portability on various UNIXes (freebsd, solaris, macosx)
stdout destination: would probably only be an SCL wrapper around the existing file driver, e.g. file("/dev/stdout");
stderr destinatiion: would probably only be an SCL wrapper around the existing file driver, e.g. file("/dev/stderr");
a means to construct the complete configuration as passed in by command line arguments; would probably need to register confgen blocks, and a larger configuration frame where they are inserted.
a means to terminate syslog-ng when we read the entire input and all outputs are finished. (would probably be the most difficult)
tests

Hope this helps.

Bazsi