[syslog-ng] diskqueue framework patch

[Balazs Scheidler]

On Tue, 2011-03-29 at 14:32 +0200, Gergely Nagy wrote:
> Balazs Scheidler <bazsi at balabit.hu> writes:
> 
> > On Mon, 2011-03-28 at 14:29 +0200, Nagy Attila wrote:
> >> Hi all,
> >> 
> >> due to the need for disk queue usage in PE 4.1 we had to extend the 
> >> syslog-ng sources, in order to be able write plugins that implement this 
> >> feature. This patch contains these modifications.
> >> Basically it extends the 'class' hierarchy (and 'virtual' functions, see 
> >> the new LogDestDriver) so that plugins can define their own queue 
> >> mechanisms (as some do this in PE modules) and these special queues can 
> >> be assigned to destinations.
> >
> > Before going into technical details, can you please elaborate on what
> > additional use of this patchset might have in the syslog-ng OSE
> > context? 
> >
> > If I understand correctly it introduces the concept of
> > per-destination-driver plugins, so that queue initialization can be
> > delegated to a plugin (e.g. disk based queueing).
> >
> > Can you perhaps shed some light on what other use-cases this could
> > solve?
> 
> Although I'm in no way involved with the patch, I've been waiting for
> this for some time now - therefore, if you don't mind, I'd like to say a
> few words why I'd love to see pluggable logqueue drivers.
> 
> To understand why I waited so eaglery for this, let me explain my
> situation: my server has very limited capabilities, only half a gig of
> ram, most of which is used by various ill-designed applications I have
> to run out of necessity, the disk is dead slow, and the available space
> is little aswell.
> 
> But all my servers (physical and virtual alike) log to this poor little
> Xen instance, because this is the only server I have permanently online
> and available. Most of the time, this isn't a big deal, as I don't
> produce all that many logs... but in rare cases, when I turn a lot of
> machines on, and they all start to do their tasks, the number of
> messages skyrockets, and it did happen that the server couldn't keep up,
> because it ran out of memory, and started to swap heavily.
> 
> Now, the good thing is, I have a very fat pipe to my central server
> (that's about the ONLY thing that is fast there), but every thing else
> (memory, disk, you name it) is scarce and slow.
> 
> Therefore, I need a solution where I can send logs to a central server,
> but the messages are stored elsewhere. Commonly, one'd use a NAS, but
> that's a bit of an overkill for my use case (not to mention that it's
> far more expensive than I would like). At the moment, I'm logging into
> MongoDB on the same server, into a capped collection in order to not run
> out of disk space (with the added catch that my setup also tries to send
> the logs to my desktop machine, if it is available, but does not fail if
> it isn't: this way, I have all the recent logs available on a central
> server, and I have most of the logs (everything but the central
> server's) on my desktop aswell, during times when it's online).
> 
> But I'd like something better, where I could queue up messages, and have
> them delivered to my desktop when it does come online. This would
> basically mean, that my queue would be online 24/7, but the storage
> would be elsewhere, on a place that is not always available.
> 
> Right now, I could only do that with a lot of dirty hacks, or with
> solutions I'm not willing to explore (increasing memory or disk space,
> or running syslog-ng PE with disk queue (but then, I still have to find
> more disk space, which is not going to happen - so PE wouldn't help me
> in this case, either)).
> 
> But if the LogQueue gains pluggability, I can write a plugin that'd
> queue messages in whatever store I want (most likely either mongodb or
> redis) and that is available, and forward it to my desktop at the first
> opportunity.
> 
> A plugin, which would loop through a few possible queues, choose the
> first available, and store the messages there temporarily, is something
> that's been the subject of many a lustful dreams.
> 
> That is my use case.
> 
> (And I wouldn't mind getting more familiar with LogQueue either, and the
> plugin-patch presents a great opportunity to do so, and statisfy my
> curiousity)

I've just pushed out the new framework for creating "inner" plugins for
source / destination drivers.

This new framework makes it possible to add functionality to already
existing source and destination drivers.

My use-case was that suppress() or perhaps throttle(), or new queueing
disciplines can be added to syslog-ng, without having to change the
source code of the corresponding drivers.

A few rules must be kept though, and I've tried to summarize them in the
commit message, and the new comments in driver.h / logpipe.h

This finally clears up some API inconsistencies and also is a good basis
to rebase my scalability patches against. (since the dropping of the
messages was delegated to LogQueue which is required for the new
scalability stuff).

It was long and tedious, got distracted a couple of times, but I'm quite
satisfied now.	

-- 
Bazsi