"Lucas, Sascha" <Sascha.Lucas@gisa.de> writes:
I'm trying to have reliable log transfer (in the sense of network failure) with syslog-ng-OSE. Here my idea (syslog-ng.conf): [...] What do you think about this solution? Are there objections?
The idea in general, is reasonably sound. The downside is that the file will grow infinitely, or until you rotate it. There are workarounds, of course, like rotating every once in a while (but then you need to make sure everything from the old file is already forwarded), but those have a couple of serious drawbacks. However, if (when) wildcard sources find their way into OSE, things will become a lot simpler, as you can easily rotate files away once in a while, and syslog-ng will close them once they stop receiving data. There was another patch on the list, that implemented size-based rotation - this needs updating & refactoring to be mergable into 3.4. If we consider a future version of syslog-ng OSE where both are merged, we could have a config like this: source s_relay { tcp(...); udp(...); }; destination d_buffer { file("/data/syslog-buffer" size-limit(1G) template("...")); }; log { source(s_relay); destination(d_buffer); }; source s_buffer { file("/data/syslog-buffer*"); }; destination d_mongo_unrel { mongodb(...); }; log { source(s_buffer); destination(d_mongo_unrel); flags(flow-control); }; You'd still need a way to clean very old files, but it's easier to do that if the buffer is not one big file.
Can someone give me a hint, how to get the actual reading position from s_buffer (i.e. from syslog-ng.persist?) and then to truncate it (shell commands preferred)?
That's not easily possible at the moment, and definitely not with shell only (ok, technically you can do it all in shell, but.. you don't want to go down that road, trust me). There is currently no easy way to inspect the contents of syslog-ng.persist, which is a shame, and a feature I've heard being asked for multiple times in the past, but noone got around to implement it yet. (I'm not even sure it would be that easy to write such a tool, but feel free to prove me wrong.) For what its worth, I'll be writing a blog post about OSE and the desire to have disk buffering, what workaround-ish solutions exist, and what will exist in the near or not so near future, if I manage to find time to write them (or bribe someone into turning various ideas and designs into working code). -- |8]