<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
I agree with you in that the syntax and the structure of patterndb
desperately needs an overhaul and I like the approach your
suggestions show. However, I've got some problems with your
proposal. See my comments below (in no particular order).<br>
<br>
1) the lisp-y syntax<br>
<br>
This is fundamentally different from the main config file of
syslog-ng and alien to those not used to do stuff in lisp-like
languages -- who I guess make up the majority of syslog-ng users.
If we are to change the format of patterndb, I'd suggest something
like what we have in the main config file or something JSON-ish,
similar to what logstash has:
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="http://logstash.net/docs/1.1.1/filters/grok">http://logstash.net/docs/1.1.1/filters/grok</a>.
At least I'd be just as annoyed to type all those parantheses as I
am typing the tons of <tags> of XML :) I want to write
patterns and not code or a huge XML. The actual container format
just needs to get out my way as much as possible.<br>
<br>
2) I like the action functions<br>
<br>
I think these are the three main operations we need
(set/clear/append), however, I wouldn't call appending conjoining
but that's just me :)<br>
<br>
3) What about pattern hierarchy == efficient matching?<br>
<br>
Your proposal allows the user to define complex conditions for a
pattern match. On the other hand, the patterns we have right now
work in a way that allows us to organize them in a radix tree and
use a greedy, non-backtracking algorithm for matching which makes
this procedure incredibly fast. Whereas if we'd allow more complex
conditions, we'd need to fall back to a linear matching: if we
have 5000 patterns, we'd have to match each and every pattern to
each incoming message. Which is slow.<br>
<br>
4) this is horrible: (match "this " (capture :qstring :as
"object") " is good")<br>
<br>
Sorry for my bluntness, but it is :) It indeed is lisp-y, but it
is hard to read and a tidious to write with all those parantheses
and "capture"s. I personally like the current syntax of the
patterns themselves and I'd keep it as it is. (Grok -- again, of
logstash fame -- also has something similar and it seems to be
working for them, too:
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a
href="https://github.com/logstash/logstash/blob/v1.0.17/patterns/linux-syslog">https://github.com/logstash/logstash/blob/v1.0.17/patterns/linux-syslog</a>)<br>
<br>
5) I agree that correlation should be handled separately -- but we
need IDs/names for that!<br>
<br>
I totally agree with you that correlation should be separated from
parsing, I always have a hard time to wrap my mind around the way
correlation works in patterndb. But to do that (and to do more
filtering or anything with this parsing), a pattern needs a name
or ID. Sure, it can be added by a (set! :pattern-name "foo")
command in your example but I think it needs a more prevalent
place.<br>
<br>
So, as a summary: I think your approach has its place but can not
and should not replace patterndb. It can be incredibly flexible
and as a result we would not have to bastardize patterndb to
support every weird use case that comes up rather simply point the
user to use these custom parsers -- but this flexibility has the
price of having to do one-by-one matching between patterns and
messages which brings in a huge performance penalty. We still have
to give an easy-to-use solution for users who simply want to write
patterns which they later use for filtering. The current XML
syntax is tidious to use, I agree, but what you suggest is, in my
opinion, even more so.<br>
<br>
greets,<br>
Peter<br>
<br>
On 09/05/2012 12:31 PM, Gergely Nagy wrote:<br>
</div>
<blockquote cite="mid:877gs8n4kw.fsf@algernon.balabit" type="cite">
<pre wrap="">In the past few months, I talked a lot about patterndb and related
things with colleagues - during coffee break, over a beer, etc -, and
last night, I got as far as drafting a proof of concept tool that
realizes some of the ideas we've had. Some of these might have been
discussed on this list too, in the past, I really can't remember all the
influences I'm afraid.
Anyway! I like the concept of patterndb, but I absolutely hate XML. It's
not a natural format for describing how to match patterns, and what to
do with them. Not for me, anyway. I'd like something that is much closer
to how I normally think, something that feels more like a programming
language, a domain specific one, engineered for this task only, but
still somewhat familiar. I also want it to be fast.
For a long time, I also wanted to play with developing both a Domain
Specific Language (DSL for short) and a compiler, but never had the
opportunity. Pattern matching is one now!
How about a small language, that we'd compile down to C, automatically
add some boilerplate, and we'd get a syslog-ng parser plugin in return?
That would mean the language is fairly easy to extend, it produces
native code, which will hopefully run as fast - if not faster - than
patterndb, and we skip the entire XML pile too!
To demonstrate, this is what I've been thinking of:
,----
| (cond :message
| (match "foo (bar) ([:number:])")
| (do
| (set! :bar "$1")
| (set! :stuff "$2")
| (conj! :tags "test stuff")))
|
| (deftest message-match
| "this is a foo bar 1234 message!"
|
| (== :bar "bar")
| (== :stuff "1234")
| (contains? :tags "test stuff"))
`----
This would compile down to roughly the following C code (with the test
excluded, for now):
,----
| gboolean
| m_something(LogMsg *message, GString *subject)
| {
| GString *m1; /* ([:number:]) */
|
| /* The subject must be at least as long as the static strings in the
| pattern, if it's shorter, we don't match */
| if (subject->len < 8)
| return FALSE;
|
| /* If we don't find part of the pattern, bail out. */
| if (strncmp(subject->str, "foo bar ") != 0)
| return FALSE;
|
| if (!find_number(subject->str + 8, &m1))
| return FALSE;
|
| /* The whole stuff matched, yay! Lets fill in the fields. */
|
| /* "bar" is a static string, fill it in as-is, no need to extract it
| from the subject. */
| log_msg_set_value(msg, "bar", "bar", 3);
|
| /* :stuff is a number, so that needs to come from the
| subject. Thankfully find_number() already did the extraction for
| us. */
| log_msg_set_value(msg, "stuff", m1->str, m1->len);
|
| log_msg_set_tag_by_name(msg, "test stuff");
|
| return TRUE;
| }
`----
I believe this is pretty efficient, the code generator can comment the
the generated source nicely too. If we add named capture-groups, then
the variables used can have meaningful names too!
For example: (match "this (?<object>[:qstring:]) is good")
In this case, the variable would be called m_object.
However, the pattern-string is a bit awkward, when the rest is lispy, it
also complicates the generator, so I was thinking of turning the pattern
into a lispy syntax too:
,----
| (match "foo " (capture "bar") " " (capture :number))
`----
Or, with named capture groups added:
,----
| (match "this " (capture :qstring :as "object") " is good")
`----
Of course, the action to take on a match can also contain another
cond+match pair, so it can be nested as deep as one wishes to, the
compiler will compile each cond into a separate function, and just call
the appropriate one. Or perhaps inline them - that's an implementation
detail, and doesn't really matter.
The big advantage I see, is a DSL that is much closer to how I think,
one that has the potential to produce a compact parser, one that is also
easy to debug with conventional tools (gdb ;) because it compiles down
to C. There is less run-time overhead too.
Also, if implemented correctly, the generator would have the parser and
the code generator entirely independent, so adding a different syntax
would be as easy as writing a parser that produces the same abstract
syntax tree the generator works with. This way, for those who're more
familiar with C-like languages, the above matcher could be rewritten
like this:
,----
| switch ($message)
| {
| case match("foo ", capture("bar"), " ", capture(:number:))
| {
| set("bar", "$1");
| set("stuff", "$2");
| append($tags, "test stuff");
| }
| }
`----
And it would compile down to the exact same C code, accompanied by an
appropriate autotools-based build system, so all you'd have to do in the
end is to write the matcher, and issue the following commands:
,----
| $ matcher-generate test-patterns.pm
| $ cd test-patterns
| $ autoreconf -i && ./configure && make && make install
`----
And finally, modify your syslog-ng.conf:
,----
| @module test-patterns
| parser p_test { parser(test-patterns); };
`----
It does have downsides, though, namely that you need to regenerate &
recompile the module and restart syslog-ng each time you modify the
source, which is less convenient than just restarting syslog-ng
itself. One also needs to learn a 'new' language to write pattern
matchers in (but one has to learn patterndb too, anyway, so this isn't
that big a disadvantage, especially since a more language-like thing is,
in my opinion, easier to learn :).
However, I believe that the advantages are worth it. For me, they
certainly do, so I already started to hash out a proof of concept. So
far, my PoC code can generate C functions, and supports a small subset
of the DSL explained below.
Do note that all this does not include corellation, because I believe
that corellation should be separate from parsing, and a similar
technique could be used to write advanced corellation setups - I will go
into detail once I have a working proof of concept compiler for the
parser.
As a start, the DSL would support the following constructs:
Top-level constructs:
---------------------
* (cond :field condition action ...)
Where :field can be any field, condition is a single condition
function (see below) and action is a single action too (see even
further below).
Any number of condition-action paris can be specified, the first one
matching will win, and the rest won't be tried. These two must always
be paired together.
* (deftest test-name
"source string"
test-conditions)
Right now, lets ignore this. But in the long run, I want to be able to
write down reasonably complex tests too. Not entirely sure yet what I
need it to do though.
Condition functions:
--------------------
* (match pattern-spec)
Matches a pattern-spec, simple as that. See below for the definition
of the pattern-spec!
* (not-match pattern-spec)
The opposite of (match): action is triggered if the pattern does not
match.
* (exists?)
Triggers the action if the field in the condition exists.
* (not-exists?)
Opposite of (exists?).
* :default or (always)
Always triggers, so one can do catch-all actions.
Action functions:
-----------------
* (set! :field value)
* (clear! :field)
These two should speak for themselves, I believe.
* (conj! :field values)
Conjoin (append) values to the specified field. Some fields (:tags) that
need to be, will be treated specially, otherwise it just appends a
separator (",") and the values.
* (do actions...)
Does all the specified actions.
Pattern spec:
-------------
The pattern can be built up from the sequence of the following things,
in any order:
* A plain string
* (capture pattern-spec [:as name])
This just marks the pattern-spec as something to capture. While the
implementation may produce code that captures things anyway, the only
guarantee that something will be available for the actions, is to wrap
it in (capture). The pattern-spec can be anything, it can contain
nested captures.
If :as name is specified, the capture will be named, and actions can
refer to the captured thing by name. Otherwise, they need to refer it
by a number. Each capture - named or anonymous - has a different
number, starting from one, increasing with each occurrence of
(capture).
* Any of the following special keywords:
* :number
* :string
* :qstring
* :ipv4-address
* :ipv6-address
* :mac-address
Future ideas
------------
Later, once the basics are ready and work, it would make sense to
introduce a way to share common blocks of code: functions and perhaps
variables.
Conclusion
==========
XML sucks, DSL rocks.
Feedback appreciated, be that on the syntax, or the initially proposed
features/functions/etc, or anything else.
</pre>
</blockquote>
<br>
</body>
</html>