<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="generator" content="Osso Notes">
<title></title></head>
<body>
<p>----- Original message -----
<br>> Hey there,
<br>> my syslog-ng has gotten quite big with 50k logs per second and the
<br>> server seems to hit the io limit at night. While a few month ago I could
<br>> run a gzip with ionice over all old logs the server doesn't like it
<br>> anymore and quite a lot of logs are storing while the compression lasts.
<br>> I'm using the ose so I've got no logstore. And for a second I've thought
<br>> about writing the logs a compressed fuse fs but... fuse :P So how are
<br>> you guys doing it?
<br>
<br>well, logstore is compressing as messages come in, but it also uses a journal that it can fall back to in case of a crash or something.
<br>
<br>on-line compression might be possible with a program('gzip') destination, however in the case of a crash (restart?) messages can get lost because of the lack of journaling: messages buffered in the pipe between syslog-ng and gzip.
<br>
<br>might help if you wrote a daily rotated buffer file, and program('gzip') in parallel, so if there's a crash, you can merge the two manually. although continuation is a problem, gzip files can't be appended to, especially if not closed cleanly.
<br>
<br>addressing the issue with hw certainly helps (write the compressed stream to a different disk), but you might buy a pe license for the price :)
<br>
<br>logstore writing scales to at least two cores, one puts messages to the journal, the other compresses & encrypts. since less data hits the disk due to compression, you use less IOPS.
<br>
<br>the logstore format itself proprietary, but we've published open source software to read its contents (see Algernon's logstore reader, can be used from java, written in closure). So it's not a secret, albeit not standardized either.
<br>
<br>hope this helps.</p>
</body>
</html>