how to archive logs efficiently
Hey there, my syslog-ng has gotten quite big with 50k logs per second and the server seems to hit the io limit at night. While a few month ago I could run a gzip with ionice over all old logs the server doesn't like it anymore and quite a lot of logs are storing while the compression lasts. I'm using the ose so I've got no logstore. And for a second I've thought about writing the logs a compressed fuse fs but... fuse :P So how are you guys doing it? -- Daniel Neubacher, Network Administrator daniel.neubacher@xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.
Daniel Neubacher <daniel.neubacher@xing.com> writes:
my syslog-ng has gotten quite big with 50k logs per second and the server seems to hit the io limit at night. While a few month ago I could run a gzip with ionice over all old logs the server doesn't like it anymore and quite a lot of logs are storing while the compression lasts.
I'm using the ose so I've got no logstore. And for a second I've thought about writing the logs a compressed fuse fs but... fuse :P So how are you guys doing it?
I've used several different approaches over the years, I'll list some of them, with pros and cons: Rotate & compress ================= The first approach I used was to simply rotate log files and compress them. This quickly killed my CPU and disks. Pros: - Simple as a brick. Cons: - CPU and IO intensive, bogs down the computer Runtime, external compression ============================= Another option I played with was to write a very small program, that accepts data on stdin, and compresses it on the fly, then I sent my logs to this destination. I also kept the most recent logs in uncompressed files too. Pros: - Fairly simple - The CPU/IO load is better spread - Uncompressed logs still available I used /var/log/FILENAME-${YEAR}${MONTH}${DAY}.log, and simply deleted old ones. - You don't need to re-read old logs to archive them, archival happens on the fly. Cons: - Requires an external program, which one will have to carefully write to not loose data. - It's much harder to reliably rotate the compressed files. My program closed the current file on SIGHUP, and opened a new one. Not too elegant, and not really configurable, but got the job done. - Still bogs down the CPU and IO. This can be partially addressed by writing the compressed files to a different disk than where you write uncompressed logs. Runtime archival to external services ===================================== Since I didn't have the resources to put anymore disks into my log server at the time, IO became a problem. So I moved the archival to a different server, by sending uncompressed logs over the network, and moving the runtime compression to the other box. This is pretty much the same solution as the one above, but instead of a local pipe, stuff is sent over the network. Pros: - Still simple - IO is done on another box, so doesn't disturb the local uncompressed log storage. Cons: - Needs a separate server - Increased network bandwidth - If archiving is slow, it can still bog down both machines due to flow control. - Needs potentially large queues on the sending side, and without disk queue, that's not the most reliable thing. Database ======== This is my current solution. I still have my local logs in files for easy access, but the archive is stored in a MongoDB cluster. Pros: - IO is spread accross a number of machines - Does not bog down the central server, ever - Structured logs, better queryability Cons: - Data is not compressed - Needs a higher amount of resources to work reliably and efficiently - More complicated to set up - MongoDB does have an overhead over simply emitting text & compressing it. A variant of this would be to use AMQP to transfer logs, then you can attach any number of archival servers onto the publisher, and spread out the work nicely. But AMQP adds its own overhead too. Other solutions =============== There's a whole lot of other ways to achieve the same thing, the above ones are only those few I've personally used in the not too distant past. -- |8]
Thanks for your detailed responses. The external compression was my last resolution but now I've maybe found my final solution. I've installed Kernel 3.2 & Btrfs with zlib compression. The fs isn't marked as stable yet but I've heard some positive responses from fellow admins. Right now I've got a real time compression rate of 84% without any cpu problems. I hope my long term test is running fine but right now I'm pretty happy. -----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Gergely Nagy Gesendet: Dienstag, 12. Februar 2013 15:10 An: Syslog-ng users' and developers' mailing list Betreff: Re: [syslog-ng] how to archive logs efficiently Daniel Neubacher <daniel.neubacher@xing.com> writes:
my syslog-ng has gotten quite big with 50k logs per second and the server seems to hit the io limit at night. While a few month ago I could run a gzip with ionice over all old logs the server doesn't like it anymore and quite a lot of logs are storing while the compression lasts.
I'm using the ose so I've got no logstore. And for a second I've thought about writing the logs a compressed fuse fs but... fuse :P So how are you guys doing it?
I've used several different approaches over the years, I'll list some of them, with pros and cons: Rotate & compress ================= The first approach I used was to simply rotate log files and compress them. This quickly killed my CPU and disks. Pros: - Simple as a brick. Cons: - CPU and IO intensive, bogs down the computer Runtime, external compression ============================= Another option I played with was to write a very small program, that accepts data on stdin, and compresses it on the fly, then I sent my logs to this destination. I also kept the most recent logs in uncompressed files too. Pros: - Fairly simple - The CPU/IO load is better spread - Uncompressed logs still available I used /var/log/FILENAME-${YEAR}${MONTH}${DAY}.log, and simply deleted old ones. - You don't need to re-read old logs to archive them, archival happens on the fly. Cons: - Requires an external program, which one will have to carefully write to not loose data. - It's much harder to reliably rotate the compressed files. My program closed the current file on SIGHUP, and opened a new one. Not too elegant, and not really configurable, but got the job done. - Still bogs down the CPU and IO. This can be partially addressed by writing the compressed files to a different disk than where you write uncompressed logs. Runtime archival to external services ===================================== Since I didn't have the resources to put anymore disks into my log server at the time, IO became a problem. So I moved the archival to a different server, by sending uncompressed logs over the network, and moving the runtime compression to the other box. This is pretty much the same solution as the one above, but instead of a local pipe, stuff is sent over the network. Pros: - Still simple - IO is done on another box, so doesn't disturb the local uncompressed log storage. Cons: - Needs a separate server - Increased network bandwidth - If archiving is slow, it can still bog down both machines due to flow control. - Needs potentially large queues on the sending side, and without disk queue, that's not the most reliable thing. Database ======== This is my current solution. I still have my local logs in files for easy access, but the archive is stored in a MongoDB cluster. Pros: - IO is spread accross a number of machines - Does not bog down the central server, ever - Structured logs, better queryability Cons: - Data is not compressed - Needs a higher amount of resources to work reliably and efficiently - More complicated to set up - MongoDB does have an overhead over simply emitting text & compressing it. A variant of this would be to use AMQP to transfer logs, then you can attach any number of archival servers onto the publisher, and spread out the work nicely. But AMQP adds its own overhead too. Other solutions =============== There's a whole lot of other ways to achieve the same thing, the above ones are only those few I've personally used in the not too distant past. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Daniel Neubacher <daniel.neubacher@xing.com> writes:
Thanks for your detailed responses. The external compression was my last resolution but now I've maybe found my final solution. I've installed Kernel 3.2 & Btrfs with zlib compression. The fs isn't marked as stable yet but I've heard some positive responses from fellow admins. Right now I've got a real time compression rate of 84% without any cpu problems. I hope my long term test is running fine but right now I'm pretty happy.
That's an interesting solution aswell, thanks for sharing it! I was already considering BTRFS for the next experimental servers I'll be bringing up, this adds one more point in its favour. :) -- |8]
Better than brtfs, maybe FreeBSD/ZFS is best option ... IMO. On Mon, Feb 18, 2013 at 3:14 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Daniel Neubacher <daniel.neubacher@xing.com> writes:
Thanks for your detailed responses. The external compression was my last resolution but now I've maybe found my final solution. I've installed Kernel 3.2 & Btrfs with zlib compression. The fs isn't marked as stable yet but I've heard some positive responses from fellow admins. Right now I've got a real time compression rate of 84% without any cpu problems. I hope my long term test is running fine but right now I'm pretty happy.
That's an interesting solution aswell, thanks for sharing it! I was already considering BTRFS for the next experimental servers I'll be bringing up, this adds one more point in its favour. :)
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You might try my ELSA project ( enterprise-log-search-and-archive.googlecode.com) which can be run in archive-only mode. It uses MySQL's archive storage engine for compression which gives you a 20:1 compression ratio and should be able to handle 50k events/sec, including proper rotation. On Mon, Feb 18, 2013 at 9:20 AM, C. L. Martinez <carlopmart@gmail.com>wrote:
Better than brtfs, maybe FreeBSD/ZFS is best option ... IMO.
On Mon, Feb 18, 2013 at 3:14 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Daniel Neubacher <daniel.neubacher@xing.com> writes:
Thanks for your detailed responses. The external compression was my last resolution but now I've maybe found my final solution. I've installed Kernel 3.2 & Btrfs with zlib compression. The fs isn't marked as stable yet but I've heard some positive responses from fellow admins. Right now I've got a real time compression rate of 84% without any cpu problems. I hope my long term test is running fine but right now I'm pretty happy.
That's an interesting solution aswell, thanks for sharing it! I was already considering BTRFS for the next experimental servers I'll be bringing up, this adds one more point in its favour. :)
-- |8]
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Unfortunately btrfs failed the test. A few days ago syslog-ng startet to do weired things. At first it only stopped logging for a few seconds and then even reloaded every 30-60 seconds with no errors logged (even with verbose & debug). After switching back to a other filesystem / a clean new btrfs partition everything worked again - 2bad. Now I'm experimenting again with the newly stable ZFS for linux :P ________________________________________ Von: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu]" im Auftrag von "Gergely Nagy [algernon@balabit.hu] Gesendet: Montag, 18. Februar 2013 16:14 An: Syslog-ng users' and developers' mailing list Betreff: Re: [syslog-ng] how to archive logs efficiently Daniel Neubacher <daniel.neubacher@xing.com> writes:
Thanks for your detailed responses. The external compression was my last resolution but now I've maybe found my final solution. I've installed Kernel 3.2 & Btrfs with zlib compression. The fs isn't marked as stable yet but I've heard some positive responses from fellow admins. Right now I've got a real time compression rate of 84% without any cpu problems. I hope my long term test is running fine but right now I'm pretty happy.
That's an interesting solution aswell, thanks for sharing it! I was already considering BTRFS for the next experimental servers I'll be bringing up, this adds one more point in its favour. :) -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
----- Original message -----
Hey there, my syslog-ng has gotten quite big with 50k logs per second and the server seems to hit the io limit at night. While a few month ago I could run a gzip with ionice over all old logs the server doesn't like it anymore and quite a lot of logs are storing while the compression lasts. I'm using the ose so I've got no logstore. And for a second I've thought about writing the logs a compressed fuse fs but... fuse :P So how are you guys doing it?
well, logstore is compressing as messages come in, but it also uses a journal that it can fall back to in case of a crash or something. on-line compression might be possible with a program('gzip') destination, however in the case of a crash (restart?) messages can get lost because of the lack of journaling: messages buffered in the pipe between syslog-ng and gzip. might help if you wrote a daily rotated buffer file, and program('gzip') in parallel, so if there's a crash, you can merge the two manually. although continuation is a problem, gzip files can't be appended to, especially if not closed cleanly. addressing the issue with hw certainly helps (write the compressed stream to a different disk), but you might buy a pe license for the price :) logstore writing scales to at least two cores, one puts messages to the journal, the other compresses & encrypts. since less data hits the disk due to compression, you use less IOPS. the logstore format itself proprietary, but we've published open source software to read its contents (see Algernon's logstore reader, can be used from java, written in closure). So it's not a secret, albeit not standardized either. hope this helps.
participants (5)
-
Balazs Scheidler
-
C. L. Martinez
-
Daniel Neubacher
-
Gergely Nagy
-
Martin Holste