[syslog-ng] [RFC]: Pattern matching & corellation ideas

[Gergely Nagy]

In the past few months, I talked a lot about patterndb and related
things with colleagues - during coffee break, over a beer, etc -, and
last night, I got as far as drafting a proof of concept tool that
realizes some of the ideas we've had. Some of these might have been
discussed on this list too, in the past, I really can't remember all the
influences I'm afraid.

Anyway! I like the concept of patterndb, but I absolutely hate XML. It's
not a natural format for describing how to match patterns, and what to
do with them. Not for me, anyway. I'd like something that is much closer
to how I normally think, something that feels more like a programming
language, a domain specific one, engineered for this task only, but
still somewhat familiar. I also want it to be fast.

For a long time, I also wanted to play with developing both a Domain
Specific Language (DSL for short) and a compiler, but never had the
opportunity. Pattern matching is one now!

How about a small language, that we'd compile down to C, automatically
add some boilerplate, and we'd get a syslog-ng parser plugin in return?
That would mean the language is fairly easy to extend, it produces
native code, which will hopefully run as fast - if not faster - than
patterndb, and we skip the entire XML pile too!

To demonstrate, this is what I've been thinking of:

,----
| (cond :message
|   (match "foo (bar) ([:number:])")
|     (do
|       (set! :bar "$1")
|       (set! :stuff "$2")
|       (conj! :tags "test stuff")))
|
| (deftest message-match
|   "this is a foo bar 1234 message!"
|
|   (== :bar "bar")
|   (== :stuff "1234")
|   (contains? :tags "test stuff"))
`----

This would compile down to roughly the following C code (with the test
excluded, for now):

,----
| gboolean
| m_something(LogMsg *message, GString *subject)
| {
|   GString *m1; /* ([:number:]) */
|
|   /* The subject must be at least as long as the static strings in the
|   pattern, if it's shorter, we don't match */
|   if (subject->len < 8)
|     return FALSE;
|
|   /* If we don't find part of the pattern, bail out. */
|   if (strncmp(subject->str, "foo bar ") != 0)
|     return FALSE;
|
|   if (!find_number(subject->str + 8, &m1))
|     return FALSE;
|
|   /* The whole stuff matched, yay! Lets fill in the fields. */
|
|   /* "bar" is a static string, fill it in as-is, no need to extract it
|   from the subject. */
|   log_msg_set_value(msg, "bar", "bar", 3);
|
|   /* :stuff is a number, so that needs to come from the
|   subject. Thankfully find_number() already did the extraction for
|   us. */
|   log_msg_set_value(msg, "stuff", m1->str, m1->len);
|
|   log_msg_set_tag_by_name(msg, "test stuff");
|
|   return TRUE;
| }
`----

I believe this is pretty efficient, the code generator can comment the
the generated source nicely too. If we add named capture-groups, then
the variables used can have meaningful names too!

For example: (match "this (?<object>[:qstring:]) is good")

In this case, the variable would be called m_object.

However, the pattern-string is a bit awkward, when the rest is lispy, it
also complicates the generator, so I was thinking of turning the pattern
into a lispy syntax too:

,----
| (match "foo " (capture "bar") " " (capture :number))
`----

Or, with named capture groups added:

,----
| (match "this " (capture :qstring :as "object") " is good")
`----

Of course, the action to take on a match can also contain another
cond+match pair, so it can be nested as deep as one wishes to, the
compiler will compile each cond into a separate function, and just call
the appropriate one. Or perhaps inline them - that's an implementation
detail, and doesn't really matter.

The big advantage I see, is a DSL that is much closer to how I think,
one that has the potential to produce a compact parser, one that is also
easy to debug with conventional tools (gdb ;) because it compiles down
to C. There is less run-time overhead too.

Also, if implemented correctly, the generator would have the parser and
the code generator entirely independent, so adding a different syntax
would be as easy as writing a parser that produces the same abstract
syntax tree the generator works with. This way, for those who're more
familiar with C-like languages, the above matcher could be rewritten
like this:

,----
| switch ($message)
|   {
|     case match("foo ", capture("bar"), " ", capture(:number:))
|       {
|         set("bar", "$1");
|         set("stuff", "$2");
|         append($tags, "test stuff");
|       }
|   }
`----

And it would compile down to the exact same C code, accompanied by an
appropriate autotools-based build system, so all you'd have to do in the
end is to write the matcher, and issue the following commands:

,----
| $ matcher-generate test-patterns.pm
| $ cd test-patterns
| $ autoreconf -i && ./configure && make && make install
`----

And finally, modify your syslog-ng.conf:

,----
| @module test-patterns
| parser p_test { parser(test-patterns); };
`----

It does have downsides, though, namely that you need to regenerate &
recompile the module and restart syslog-ng each time you modify the
source, which is less convenient than just restarting syslog-ng
itself. One also needs to learn a 'new' language to write pattern
matchers in (but one has to learn patterndb too, anyway, so this isn't
that big a disadvantage, especially since a more language-like thing is,
in my opinion, easier to learn :).

However, I believe that the advantages are worth it. For me, they
certainly do, so I already started to hash out a proof of concept. So
far, my PoC code can generate C functions, and supports a small subset
of the DSL explained below.

Do note that all this does not include corellation, because I believe
that corellation should be separate from parsing, and a similar
technique could be used to write advanced corellation setups - I will go
into detail once I have a working proof of concept compiler for the
parser.

As a start, the DSL would support the following constructs:

Top-level constructs:
---------------------

* (cond :field condition action ...)

  Where :field can be any field, condition is a single condition
  function (see below) and action is a single action too (see even
  further below).

  Any number of condition-action paris can be specified, the first one
  matching will win, and the rest won't be tried. These two must always
  be paired together.

* (deftest test-name
    "source string"

    test-conditions)

  Right now, lets ignore this. But in the long run, I want to be able to
  write down reasonably complex tests too. Not entirely sure yet what I
  need it to do though.

Condition functions:
--------------------

* (match pattern-spec)

  Matches a pattern-spec, simple as that. See below for the definition
  of the pattern-spec!

* (not-match pattern-spec)

  The opposite of (match): action is triggered if the pattern does not
  match.

* (exists?)

  Triggers the action if the field in the condition exists.

* (not-exists?)

  Opposite of (exists?).

* :default or (always)

  Always triggers, so one can do catch-all actions.

Action functions:
-----------------

* (set! :field value)
* (clear! :field)

These two should speak for themselves, I believe.

* (conj! :field values)

Conjoin (append) values to the specified field. Some fields (:tags) that
need to be, will be treated specially, otherwise it just appends a
separator (",") and the values.

* (do actions...)

Does all the specified actions.

Pattern spec:
-------------

The pattern can be built up from the sequence of the following things,
in any order:

* A plain string

* (capture pattern-spec [:as name])

  This just marks the pattern-spec as something to capture. While the
  implementation may produce code that captures things anyway, the only
  guarantee that something will be available for the actions, is to wrap
  it in (capture). The pattern-spec can be anything, it can contain
  nested captures.

  If :as name is specified, the capture will be named, and actions can
  refer to the captured thing by name. Otherwise, they need to refer it
  by a number. Each capture - named or anonymous - has a different
  number, starting from one, increasing with each occurrence of
  (capture).

* Any of the following special keywords:

  * :number
  * :string
  * :qstring
  * :ipv4-address
  * :ipv6-address
  * :mac-address

Future ideas
------------

Later, once the basics are ready and work, it would make sense to
introduce a way to share common blocks of code: functions and perhaps
variables.

Conclusion
==========

XML sucks, DSL rocks.

Feedback appreciated, be that on the syntax, or the initially proposed
features/functions/etc, or anything else.

-- 
|8],
who perhaps spent a little too much time near LISP-y stuff