Balazs Scheidler <bazsi@balabit.hu> writes:
On Mon, 2011-03-28 at 14:29 +0200, Nagy Attila wrote:
Hi all,
due to the need for disk queue usage in PE 4.1 we had to extend the syslog-ng sources, in order to be able write plugins that implement this feature. This patch contains these modifications. Basically it extends the 'class' hierarchy (and 'virtual' functions, see the new LogDestDriver) so that plugins can define their own queue mechanisms (as some do this in PE modules) and these special queues can be assigned to destinations.
Before going into technical details, can you please elaborate on what additional use of this patchset might have in the syslog-ng OSE context?
If I understand correctly it introduces the concept of per-destination-driver plugins, so that queue initialization can be delegated to a plugin (e.g. disk based queueing).
Can you perhaps shed some light on what other use-cases this could solve?
Although I'm in no way involved with the patch, I've been waiting for this for some time now - therefore, if you don't mind, I'd like to say a few words why I'd love to see pluggable logqueue drivers. To understand why I waited so eaglery for this, let me explain my situation: my server has very limited capabilities, only half a gig of ram, most of which is used by various ill-designed applications I have to run out of necessity, the disk is dead slow, and the available space is little aswell. But all my servers (physical and virtual alike) log to this poor little Xen instance, because this is the only server I have permanently online and available. Most of the time, this isn't a big deal, as I don't produce all that many logs... but in rare cases, when I turn a lot of machines on, and they all start to do their tasks, the number of messages skyrockets, and it did happen that the server couldn't keep up, because it ran out of memory, and started to swap heavily. Now, the good thing is, I have a very fat pipe to my central server (that's about the ONLY thing that is fast there), but every thing else (memory, disk, you name it) is scarce and slow. Therefore, I need a solution where I can send logs to a central server, but the messages are stored elsewhere. Commonly, one'd use a NAS, but that's a bit of an overkill for my use case (not to mention that it's far more expensive than I would like). At the moment, I'm logging into MongoDB on the same server, into a capped collection in order to not run out of disk space (with the added catch that my setup also tries to send the logs to my desktop machine, if it is available, but does not fail if it isn't: this way, I have all the recent logs available on a central server, and I have most of the logs (everything but the central server's) on my desktop aswell, during times when it's online). But I'd like something better, where I could queue up messages, and have them delivered to my desktop when it does come online. This would basically mean, that my queue would be online 24/7, but the storage would be elsewhere, on a place that is not always available. Right now, I could only do that with a lot of dirty hacks, or with solutions I'm not willing to explore (increasing memory or disk space, or running syslog-ng PE with disk queue (but then, I still have to find more disk space, which is not going to happen - so PE wouldn't help me in this case, either)). But if the LogQueue gains pluggability, I can write a plugin that'd queue messages in whatever store I want (most likely either mongodb or redis) and that is available, and forward it to my desktop at the first opportunity. A plugin, which would loop through a few possible queues, choose the first available, and store the messages there temporarily, is something that's been the subject of many a lustful dreams. That is my use case. (And I wouldn't mind getting more familiar with LogQueue either, and the plugin-patch presents a great opportunity to do so, and statisfy my curiousity) -- |8]