No subject

close to C/C++ speeds because they're essentially using the same
library and therefore mostly the same syscalls.  I'd be really
interested if anyone has benchmarks.  I'd expect something like 10%
better performance in C, but not much more, assuming that the vast
majority of CPU time is spent on PCRE.

> Personally, I'd make the last step routing back into syslog-ng with a sou=
rce on a custom port and letting syslog handle the writing to disk. That wa=
y you can still use macros such as timestamps, etc.. Then again, that may b=
e because I do that all the time. ;) A log statement that takes everything =
from the custom source and logs to a file should work beautifully; no need =
for filters though you could still do additional processing if needed. That=
 said I'd also consider running a daemon that accepted all the input, forma=
tted it, and then sent it to syslog-ng, pointing the clients at the custom =
daemon if that was possible.
>
> One advantage to the daemon route is that it wouldn't *have* to reside on=
 the same system.

Yep, you could definitely let Syslog-NG handle the last mile as well.
I was trying to keep the scope as narrow as possible in my example.

I wonder if you could build an NFA state machine by conditionally
looping output from a pattern-db parsed message into a source in
Syslog-NG with a different pattern-db, depending on the previous
output.  Something like a token parser pdb that does an ESTRING up
until " " and another one that only expects the key/val pair to be
sent to it as the message.  So it comes in as k1=3Dv1 k2=3Dv2 and the
first kv gets gobbled up and then sent to another pdb source with a
pdb which only matches if the message starts with certain terms.  Then
the rest of the original message is looped back to itself using
@ANYSTRING@ to capture the remainder, that is, minus the kv which was
sent to the kv pdb.  It would keep recursively looping like that until
there's no message left.  If that all worked, your pattern db would be
extremely simple as it would just be a pattern per key you were
looking for, and order would no longer be an issue.  Of course there's
still the problem of demuxing the whole thing back into a coherent
message, but I think that could be done a number of ways by passing
the MSGID token with each part and using the new conditionals present
in OSE 3.2.  If OSE 3.3 can really do close to 1 million msgs/sec,
then the overhead of resubmitting the same log many times may be
bearable, especially with the threading.