reliable log transfer with syslog-ng-OSE (a bit like PEs' diskbuffering)
Hi, I'm trying to have reliable log transfer (in the sense of network failure) with syslog-ng-OSE. Here my idea (syslog-ng.conf): ## version='3.3.5' ## first save every message to a file source s_relay { tcp( max-connections(10) port(4321) ); udp( port(4321) ); }; destination d_buffer { file("/data/syslog-buffer" template("$ISODATE $HOST $SOURCEIP $PROGRAM $PID $PRI $MSG\n") ); }; log { source(s_relay); destination(d_buffer); }; ## then follow this file and parse it with flow-control enabled source s_buffer { file("/data/syslog-buffer" flags(no-parse) follow_freq(0.1)); }; parser p_buffer { csv-parser( columns("buffer.isodate", "buffer.host", "buffer.srcip", "buffer.program", "buffer.pid", "buffer.pri", "buffer.msg") delimiters(" ") flags(greedy) ); }; destination d_mongo_unrel { mongodb(...); }; log { source(s_buffer); parser(p_buffer); destination(d_mongo_unrel); flags(flow-control); }; What do you think about this solution? Are there objections? I need to add follow_freq() in s_buffer. Without it, the mongodb fills a bit slow. But I don't understand why. Can someone give me a hint, how to get the actual reading position from s_buffer (i.e. from syslog-ng.persist?) and then to truncate it (shell commands preferred)? TIA, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.
"Lucas, Sascha" <Sascha.Lucas@gisa.de> writes:
I'm trying to have reliable log transfer (in the sense of network failure) with syslog-ng-OSE. Here my idea (syslog-ng.conf): [...] What do you think about this solution? Are there objections?
The idea in general, is reasonably sound. The downside is that the file will grow infinitely, or until you rotate it. There are workarounds, of course, like rotating every once in a while (but then you need to make sure everything from the old file is already forwarded), but those have a couple of serious drawbacks. However, if (when) wildcard sources find their way into OSE, things will become a lot simpler, as you can easily rotate files away once in a while, and syslog-ng will close them once they stop receiving data. There was another patch on the list, that implemented size-based rotation - this needs updating & refactoring to be mergable into 3.4. If we consider a future version of syslog-ng OSE where both are merged, we could have a config like this: source s_relay { tcp(...); udp(...); }; destination d_buffer { file("/data/syslog-buffer" size-limit(1G) template("...")); }; log { source(s_relay); destination(d_buffer); }; source s_buffer { file("/data/syslog-buffer*"); }; destination d_mongo_unrel { mongodb(...); }; log { source(s_buffer); destination(d_mongo_unrel); flags(flow-control); }; You'd still need a way to clean very old files, but it's easier to do that if the buffer is not one big file.
Can someone give me a hint, how to get the actual reading position from s_buffer (i.e. from syslog-ng.persist?) and then to truncate it (shell commands preferred)?
That's not easily possible at the moment, and definitely not with shell only (ok, technically you can do it all in shell, but.. you don't want to go down that road, trust me). There is currently no easy way to inspect the contents of syslog-ng.persist, which is a shame, and a feature I've heard being asked for multiple times in the past, but noone got around to implement it yet. (I'm not even sure it would be that easy to write such a tool, but feel free to prove me wrong.) For what its worth, I'll be writing a blog post about OSE and the desire to have disk buffering, what workaround-ish solutions exist, and what will exist in the near or not so near future, if I manage to find time to write them (or bribe someone into turning various ideas and designs into working code). -- |8]
participants (2)
-
Gergely Nagy
-
Lucas, Sascha