[syslog-ng]Filter enhancement

[Brad Arlt]

On Thu, Dec 06, 2001 at 01:30:15PM +0100, Gregor Binder wrote:
> Jay Guerette on Thu, Dec 06, 2001 at 01:22:58AM -0500:
> You would need two source definitions, one for "regular" message trans-
> port to syslog-ng (would probably contain internal, /dev/log, etc. and
> your network port(s)), and the other one for messages that are fed back
> to syslog-ng (which IMO should be a FIFO, because that gives you more
> flexibility when choosing a language to implement your external filter
> programs).
> 
> The log statement for the first source would then probably be so that
> all syslog-ng filters ("internal" filters in that case) would be applied
> to the log message, while the log statement for the second source
> definition could possibly be without any filters, and directly going to
> a specific log.
> 
> What I was trying to say was: Since you can specify multiple sources,
> and also which filters get applied to what source, you can get the
> functionality you ask for without IMHO having too much overhead vs. the
> solution you suggest. You'd obviously have to optimize the configuration
> suitably to your environment.
> 
> I hope I made myself clear this time .. :)

Pardon me while the mail admin within slips out ;)

What you are descibing is not too dissimilar from Milter, the mail
filter API in Sendmail.  You would write programs that listen on a
AF_UNIX or AF_INET socket, sendmail connects and talks over the socket
and your filter says "accect", "continue filtering", "reject but try
again later", and "reject".

Now this works fine for email, as the message is large and overhead,
while a concern, is not that big a deal.  In this case you are using a
pipe file, not a socket, but there is still the issue syslog-ng seeing
some messages twice.

A modification of your idea would get around this however.  If you split
the tasks of filtering, receiving a message, and writing as message we
can setup a pipeline (and allow scalling accross CPUs).

This should give you an model that can have an arbitrary number of
filters, as filters could take input and give output to other filters.
It would also give the speed concious the ability to have no filters
at all (beyond the status quo in syslog-ng maybe).

Another advantage to this approach is your message writing (or delivering)
program/process could take any form you desired.  Files, network sockects,
database, and pagers could all have an output program.

The downside of this is filtering for different output "devices" would
be difficult (basic priority/facilty/program mappings would be easy,
more exotic would not (or atleast I can think of how to do so easy
this close to going to the bar)).

--------

My brain informs me that perhaps instead of the same syslog-ng process
seeing each message, you could do as I suggest with syslog-ng as the
lister and the delivery agent connected via pipes that snake through your
filter programs.  This would let you throw CPU power at the problem,
but so would threading I suppose.

--------

All of this hinges on your OS scaling well in a multiprocessor
environment.  Linux 2.4 isn't bad, but it still kinda sucks.  I have a
feeling that with this and other solutions using pipes bottle necks
will form at kernel.  For those running Solaris and Tru64 (maybe AIX
and IRIX) this will be less of a problem.  I shan't comment on *BSD.

Sockets might also scale better despite their appearance of sloth
(setup time only mostly).

I have a feeling that in high load environment that need fancy
filtering to happen real time (and not offline which removes any need
for making syslog-ng any more complicated than it is), threads and
plugins (shared object libraries are likely the best choice) are the
way to go.  It would let more than one CPU have a crack at filtering
and message processing, and it would not rely on the kernel to
play nice with pipes.

Ah well, that's my two cents.
----------------------------------------------------------------------------
   __o		Bradley Arlt				Security Team Lead
 _ \<_		arlt@cpsc.ucalgary.ca			University Of Calgary
(_)/(_) 	http://pages.cpsc.ucalgary.ca/~arlt/	Computer Science