[syslog-ng] reliable log transfer with syslog-ng-OSE (a bit like PEs' diskbuffering)

[Gergely Nagy]

"Lucas, Sascha" <Sascha.Lucas at gisa.de> writes:

> I'm trying to have reliable log transfer (in the sense of network
> failure) with syslog-ng-OSE. Here my idea (syslog-ng.conf):
[...]
> What do you think about this solution? Are there objections?

The idea in general, is reasonably sound. The downside is that the file
will grow infinitely, or until you rotate it. There are workarounds, of
course, like rotating every once in a while (but then you need to make
sure everything from the old file is already forwarded), but those have
a couple of serious drawbacks.

However, if (when) wildcard sources find their way into OSE, things will
become a lot simpler, as you can easily rotate files away once in a
while, and syslog-ng will close them once they stop receiving data.

There was another patch on the list, that implemented size-based
rotation - this needs updating & refactoring to be mergable into 3.4.

If we consider a future version of syslog-ng OSE where both are merged,
we could have a config like this:

source s_relay { tcp(...); udp(...); };
destination d_buffer {
            file("/data/syslog-buffer" size-limit(1G) template("..."));
};
log { source(s_relay); destination(d_buffer); };

source s_buffer { file("/data/syslog-buffer*"); };
destination d_mongo_unrel { mongodb(...); };
log { source(s_buffer); destination(d_mongo_unrel); flags(flow-control); };

You'd still need a way to clean very old files, but it's easier to do
that if the buffer is not one big file.

> Can someone give me a hint, how to get the actual reading position
> from s_buffer (i.e. from syslog-ng.persist?) and then to truncate it
> (shell commands preferred)?

That's not easily possible at the moment, and definitely not with shell
only (ok, technically you can do it all in shell, but.. you don't want
to go down that road, trust me). There is currently no easy way to
inspect the contents of syslog-ng.persist, which is a shame, and a
feature I've heard being asked for multiple times in the past, but noone
got around to implement it yet. (I'm not even sure it would be that easy
to write such a tool, but feel free to prove me wrong.)

For what its worth, I'll be writing a blog post about OSE and the desire
to have disk buffering, what workaround-ish solutions exist, and what
will exist in the near or not so near future, if I manage to find time
to write them (or bribe someone into turning various ideas and designs
into working code).

-- 
|8]