[syslog-ng] diskqueue framework patch

[Gergely Nagy]

Balazs Scheidler <bazsi at balabit.hu> writes:

> On Mon, 2011-03-28 at 14:29 +0200, Nagy Attila wrote:
>> Hi all,
>> 
>> due to the need for disk queue usage in PE 4.1 we had to extend the 
>> syslog-ng sources, in order to be able write plugins that implement this 
>> feature. This patch contains these modifications.
>> Basically it extends the 'class' hierarchy (and 'virtual' functions, see 
>> the new LogDestDriver) so that plugins can define their own queue 
>> mechanisms (as some do this in PE modules) and these special queues can 
>> be assigned to destinations.
>
> Before going into technical details, can you please elaborate on what
> additional use of this patchset might have in the syslog-ng OSE
> context? 
>
> If I understand correctly it introduces the concept of
> per-destination-driver plugins, so that queue initialization can be
> delegated to a plugin (e.g. disk based queueing).
>
> Can you perhaps shed some light on what other use-cases this could
> solve?

Although I'm in no way involved with the patch, I've been waiting for
this for some time now - therefore, if you don't mind, I'd like to say a
few words why I'd love to see pluggable logqueue drivers.

To understand why I waited so eaglery for this, let me explain my
situation: my server has very limited capabilities, only half a gig of
ram, most of which is used by various ill-designed applications I have
to run out of necessity, the disk is dead slow, and the available space
is little aswell.

But all my servers (physical and virtual alike) log to this poor little
Xen instance, because this is the only server I have permanently online
and available. Most of the time, this isn't a big deal, as I don't
produce all that many logs... but in rare cases, when I turn a lot of
machines on, and they all start to do their tasks, the number of
messages skyrockets, and it did happen that the server couldn't keep up,
because it ran out of memory, and started to swap heavily.

Now, the good thing is, I have a very fat pipe to my central server
(that's about the ONLY thing that is fast there), but every thing else
(memory, disk, you name it) is scarce and slow.

Therefore, I need a solution where I can send logs to a central server,
but the messages are stored elsewhere. Commonly, one'd use a NAS, but
that's a bit of an overkill for my use case (not to mention that it's
far more expensive than I would like). At the moment, I'm logging into
MongoDB on the same server, into a capped collection in order to not run
out of disk space (with the added catch that my setup also tries to send
the logs to my desktop machine, if it is available, but does not fail if
it isn't: this way, I have all the recent logs available on a central
server, and I have most of the logs (everything but the central
server's) on my desktop aswell, during times when it's online).

But I'd like something better, where I could queue up messages, and have
them delivered to my desktop when it does come online. This would
basically mean, that my queue would be online 24/7, but the storage
would be elsewhere, on a place that is not always available.

Right now, I could only do that with a lot of dirty hacks, or with
solutions I'm not willing to explore (increasing memory or disk space,
or running syslog-ng PE with disk queue (but then, I still have to find
more disk space, which is not going to happen - so PE wouldn't help me
in this case, either)).

But if the LogQueue gains pluggability, I can write a plugin that'd
queue messages in whatever store I want (most likely either mongodb or
redis) and that is available, and forward it to my desktop at the first
opportunity.

A plugin, which would loop through a few possible queues, choose the
first available, and store the messages there temporarily, is something
that's been the subject of many a lustful dreams.

That is my use case.

(And I wouldn't mind getting more familiar with LogQueue either, and the
plugin-patch presents a great opportunity to do so, and statisfy my
curiousity)

-- 
|8]