----- Original message -----
Hey there, my syslog-ng has gotten quite big with 50k logs per second and the server seems to hit the io limit at night. While a few month ago I could run a gzip with ionice over all old logs the server doesn't like it anymore and quite a lot of logs are storing while the compression lasts. I'm using the ose so I've got no logstore. And for a second I've thought about writing the logs a compressed fuse fs but... fuse :P So how are you guys doing it?
well, logstore is compressing as messages come in, but it also uses a journal that it can fall back to in case of a crash or something. on-line compression might be possible with a program('gzip') destination, however in the case of a crash (restart?) messages can get lost because of the lack of journaling: messages buffered in the pipe between syslog-ng and gzip. might help if you wrote a daily rotated buffer file, and program('gzip') in parallel, so if there's a crash, you can merge the two manually. although continuation is a problem, gzip files can't be appended to, especially if not closed cleanly. addressing the issue with hw certainly helps (write the compressed stream to a different disk), but you might buy a pe license for the price :) logstore writing scales to at least two cores, one puts messages to the journal, the other compresses & encrypts. since less data hits the disk due to compression, you use less IOPS. the logstore format itself proprietary, but we've published open source software to read its contents (see Algernon's logstore reader, can be used from java, written in closure). So it's not a secret, albeit not standardized either. hope this helps.