[syslog-ng] Logging all message metadata

[Balazs Scheidler]

On Mon, 2010-10-25 at 23:51 -0400, Lars Kellogg-Stedman wrote:
> > though only for a finite number of fields.  If you use generic names
> > for your extractions "@NUMBER:i0:@ @NUMBER:i1:@ @ESTRING:s0:%@ etc.
> > then your single template works for any message:
> 
> I understood the suggestion.
> 
> This is the point I'm trying to make: If I'm using, for example, the
> community patterndb database, then the metadata includes named values
> (e.g., "flowevt.src_ip") that I may not be aware of in advance.
> Furthermore, the values associated with a given class may change as
> the pattern database changes over time.  This will inherently break
> any sort of positional schema.
> 
> I am looking for way to extract all of the metadata names and values
> known to syslog-ng at the time the message is logged.  I'm not wedded
> to a database solution; if I could generate a structured output format
> like XML or JSON I could obviously post-process in whatever fashion
> best suited my needs.
> 
> I'm currently poking around the source to see if I can figure out how
> to do this.

This is exactly the way to go forward. The functionality you are looking
for is "template functions".

What I had in mind was to write a set of "format-xxx" functions, where
xxx would denote a common logging format.

One of these logging format I know is "WELF", which is simply:

name1=value1 name2=value2

and so on.

If "value" contains a space, it is enclosed in quotes. If it contains
spaces and quotes, I'm not sure what happens. I haven't found an
appropriate documentation on that.

But anyway, what we could write is a format-welf template function which
would be capable of writing out a set of NV pairs or the complete list,
customizable by parameters.

Since a template function is quite similar to a UNIX shell expansion (it
uses the bash syntax for that), it is possible to use "command-line
arguments" to specify what you would like to do:

syntax:
  $(format-welf [options] name[=value-expr]...)

Behaviour:
  The format-welf function emits a set of name-value pairs according to
the WebTrends Enhanced Log Format. In order to specify which name-value
pairs are written please specify them explicitly on the command line, or
use the --select command line option.

Command line options:
  --select <glob expression>   Specify which name-value pairs are
included in the result. The parameter is a shell-like glob pattern.
  --all                        Equivalent to '--select *'
  --prefix <string>            All name-value pairs should be emitted
with a prefix.
  --ltrim <string>             Remove <string> from the beginning of the
name-value pair _before_ adding the prefix.
  --skip-builtin               Don't include built-in name-value pairs
(e.g. the ones in the syslog header)

Arguments:
  The list of arguments after the options can specify which name-value
pairs are to be included. The expected format for each argument is
name[=value-expr], where "name" specifies the WELF identifier for the
field, and the optional value-expr is a quote-enclosed syslog-ng
template string. In case the value part is missing, the "name" will be
used as the name of the syslog-ng name-value pair. The quotes are only
included in the result if the content of the nv pair would cause the
WELF format to be ambigous.

Examples:

$(format-welf foo bar)
  Becomes: foo=FOO bar=BAR assuming foo contains the value FOO, and bar
contains BAR.

$(format-welf time="$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC" src="$SOURCEIP")
  Becomes: time="2010-10-28 21:05:53" src=1.2.3.4

$(format-welf --select .SDATA.meta.* --prefix meta --ltrim ".SDATA.meta")
  Becomes: meta.sequenceId=5 meta.tzKnown=1

What do you think? Would you like to implement such a functionality? I'd
love to include that in the convertfuncs module in 3.2

-- 
Bazsi