syslog-ng log file rotation based on size
Many of us run syslog-ng on systems with a 2 gig file size limit. An attacker can flood our loghost until logs reach 2 gigs in size, at which point syslog-ng cannot write to the file any further. If you run cfengine on your loghost, you can make it rotate logs based on size: -------------------------------------------------------- disable: loghost:: /var/log/messages size=>2000mb rotate=100 define=syslog_rotated /var/log/syslog size=>2000mb rotate=100 define=syslog_rotated processes: loghost.syslog_rotated:: "syslog-ng" signal=hup shellcommands: loghost.syslog_rotated:: "/bin/echo a log file in /var/log was rotated to avoid the 2 gig file limit on loghost|/usr/bin/mail -s'cfengine did an emergency syslog log rotation on loghost' oncall@example" -------------------------------------------------------- I'm sure there's other out of band mechanisms that could be used (a shell script whipped up in a few minutes could do it, run from cron), but it's important to think about it. Even if you have huge disks, if your OS/filesystem/file utilities can't handle large files you're in trouble. Perhaps down the road syslog-ng could include a feature to help with this. It doesn't eliminate the threat of attackers flooding your logs, but at least it could make every effort to keep from losing logs (until your filesystem fills up or the host's CPU/memory resources get overloaded under the flood, I know this isn't perfect). Even if the host OS can handle huge files, from an administration standpoint it's sometimes best to keep them smaller and deal with them in smaller chunks anyways; especially when you need to rotate logs off a machine to save local disk space. Any hope of such a feature Bazsi? Perhaps in syslog-ng 2? -- Nate Campi http://www.campin.net The porcupine with the sharpest quills gets stuck on a tree more often.
The same functionality exist in logrotate, and is a standard part of many Linux OS's. Richard
-----Original Message----- From: syslog-ng-admin@lists.balabit.hu [mailto:syslog-ng-admin@lists.balabit.hu] On Behalf Of Nate Campi Sent: Saturday, February 01, 2003 10:33 AM To: syslog-ng@lists.balabit.hu Subject: [syslog-ng]syslog-ng log file rotation based on size
Many of us run syslog-ng on systems with a 2 gig file size limit. An attacker can flood our loghost until logs reach 2 gigs in size, at which point syslog-ng cannot write to the file any further.
If you run cfengine on your loghost, you can make it rotate logs based on size:
-------------------------------------------------------- disable: loghost:: /var/log/messages size=>2000mb rotate=100 define=syslog_rotated /var/log/syslog size=>2000mb rotate=100 define=syslog_rotated
processes: loghost.syslog_rotated:: "syslog-ng" signal=hup
shellcommands: loghost.syslog_rotated:: "/bin/echo a log file in /var/log was rotated to avoid the 2 gig file limit on loghost|/usr/bin/mail -s'cfengine did an emergency syslog log rotation on loghost' oncall@example"
--------------------------------------------------------
I'm sure there's other out of band mechanisms that could be used (a shell script whipped up in a few minutes could do it, run from cron), but it's important to think about it. Even if you have huge disks, if your OS/filesystem/file utilities can't handle large files you're in trouble.
Perhaps down the road syslog-ng could include a feature to help with this. It doesn't eliminate the threat of attackers flooding your logs, but at least it could make every effort to keep from losing logs (until your filesystem fills up or the host's CPU/memory resources get overloaded under the flood, I know this isn't perfect).
Even if the host OS can handle huge files, from an administration standpoint it's sometimes best to keep them smaller and deal with them in smaller chunks anyways; especially when you need to rotate logs off a machine to save local disk space.
Any hope of such a feature Bazsi? Perhaps in syslog-ng 2? -- Nate Campi http://www.campin.net
The porcupine with the sharpest quills gets stuck on a tree more often.
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/sysl> og-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Sat, Feb 01, 2003 at 10:33:03AM -0800, Nate Campi wrote:
standpoint it's sometimes best to keep them smaller and deal with them in smaller chunks anyways; especially when you need to rotate logs off a machine to save local disk space.
Any hope of such a feature Bazsi? Perhaps in syslog-ng 2?
I don't really want to bloat syslog-ng too much with log rotation, unless the feature is general enough. The possibility to call an external program regularly for each file destination would be fine, however how often should this program be called? Or create three options like: rotate_program rotate_if_older rotate_if_larger And pass the file to be rotated to rotate_program? rotate_if_older would be problematic as syslog-ng has no information when it started to write to a given file. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
--On Sunday, February 02, 2003 12:14 PM +0100 Balazs Scheidler <bazsi@balabit.hu> wrote:
rotate_if_older would be problematic as syslog-ng has no information when it started to write to a given file.
stat() will happily tell you the file creation time. -- Carson
On 2003-02-02 06:38:49 -0500, Carson Gaspar wrote:
--On Sunday, February 02, 2003 12:14 PM +0100 Balazs Scheidler <bazsi@balabit.hu> wrote:
rotate_if_older would be problematic as syslog-ng has no information when it started to write to a given file.
stat() will happily tell you the file creation time.
Not on Unix. st_ctime is the time of last change, not the creation time. hp -- _ | Peter J. Holzer | To a database person, |_|_) | Sysadmin WSR | every nail looks like a thumb. | | | hjp@hjp.at | __/ | http://www.hjp.at/ | -- Jamie Zawinski
--On Sunday, February 02, 2003 1:17 PM +0100 "Peter J. Holzer" <hjp@hjp.at> wrote:
On 2003-02-02 06:38:49 -0500, Carson Gaspar wrote:
stat() will happily tell you the file creation time.
Not on Unix. st_ctime is the time of last change, not the creation time.
Pardon me - I was clearly on crack when I wrote that. Yes, standard UFS/FFS only maintain modified and accessed times, not creation times. Ah well. -- Carson
I really do not think that it is necessary to add this to a logging program. There are other features that I would prefer to see before this. There are a lot of other programs that already have this functionality for syslog-ng. If you run these often enough you should not have your log size problem. Richard
-----Original Message----- From: syslog-ng-admin@lists.balabit.hu [mailto:syslog-ng-admin@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Sunday, February 02, 2003 3:15 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng]syslog-ng log file rotation based on size
On Sat, Feb 01, 2003 at 10:33:03AM -0800, Nate Campi wrote:
standpoint it's sometimes best to keep them smaller and deal with them in smaller chunks anyways; especially when you need to rotate logs off a machine to save local disk space.
Any hope of such a feature Bazsi? Perhaps in syslog-ng 2?
I don't really want to bloat syslog-ng too much with log rotation, unless the feature is general enough.
The possibility to call an external program regularly for each file destination would be fine, however how often should this program be called?
Or create three options like:
rotate_program rotate_if_older rotate_if_larger
And pass the file to be rotated to rotate_program?
rotate_if_older would be problematic as syslog-ng has no information when it started to write to a given file.
-- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/sysl> og-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Sun, Feb 02, 2003 at 08:09:15AM -0800, Richard E. Perlotto II wrote:
I really do not think that it is necessary to add this to a logging program. There are other features that I would prefer to see before this. There are a lot of other programs that already have this functionality for syslog-ng. If you run these often enough you should not have your log size problem.
It's not mandatory in syslog-ng, I just want to discuss ways of addressing the issue of file size. You'll notice I already gave suggestions on perfectly good ways to manage the file sizes (complete with a working example cfengine config). It's simply that we already have a program that knows which logs are used, is already running, and is written/maintained by a trusted author (syslog-ng). Giving syslog-ng users a method to address this, when many aren't in a position to write their own utilities, or address it on their own, is a good thing. I see it as a matter of reducing complexity for the user population, not for me personally. Hmm, I fully expect this idea to be rejected for a good reason, the bloat issue...but I wanted to see it discussed. -- Nate Campi http://www.campin.net "A computer lets you make more mistakes faster than any invention in human history - with the possible exceptions of handguns and tequila." -Mitch Ratliffe "Never trust a computer you can't throw out a window." -Steve Wozniak
Heh, did not mean to sound short. You are correct in that, and what you gave was one good example. I would like to see syslog-ng focused on what it's primary purpose is. Which is able to be a flexible logging mechanism that is much superior to regular Syslog. Bloat is only one reason to not have this functionality, but as I said there is a lot of other functionality that would be good to have before we start looking at taking over the functionality of other programs as well. One thing I would really like to see is the capability to breakdown the Syslog message itself into smaller chunks such that I do not have to put it through yet another interrupter. Although I would be concerned with impacting the speed of the application then. As an example, there are a lot of logging devices out there and the content of those logs are not always consistent. Before writing to a database (or buffer file), I would like to make the logs a bit more homogeneous for usefulness. For example, the differences between what I get from the logs from Cisco devices verses Shorewall (IPTables). There is a lot of extra information that is contained in each of those logs. Not all of it do I care about. It would be really nice to be able to take certain specific pieces and only track those in a database. The ability to write to a PGP encrypted file would be excellent. Although I am not sure how this would be best implemented. The idea of built-in SSL is another that would be nice to have. While stunnel works, it is only one more tack-on that could break when you are expecting your logs to arrive somewhere. I am sure that there are many other features that many of us would like to see as well. Richard
-----Original Message----- From: syslog-ng-admin@lists.balabit.hu [mailto:syslog-ng-admin@lists.balabit.hu] On Behalf Of Nate Campi Sent: Sunday, February 02, 2003 10:54 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng]syslog-ng log file rotation based on size
On Sun, Feb 02, 2003 at 08:09:15AM -0800, Richard E. Perlotto II wrote:
I really do not think that it is necessary to add this to a logging program. There are other features that I would prefer to see before this. There are a lot of other programs that already have this functionality for syslog-ng. If you run these often enough you should not have your log size problem.
It's not mandatory in syslog-ng, I just want to discuss ways of addressing the issue of file size. You'll notice I already gave suggestions on perfectly good ways to manage the file sizes (complete with a working example cfengine config).
It's simply that we already have a program that knows which logs are used, is already running, and is written/maintained by a trusted author (syslog-ng). Giving syslog-ng users a method to address this, when many aren't in a position to write their own utilities, or address it on their own, is a good thing. I see it as a matter of reducing complexity for the user population, not for me personally.
Hmm, I fully expect this idea to be rejected for a good reason, the bloat issue...but I wanted to see it discussed. -- Nate Campi http://www.campin.net
"A computer lets you make more mistakes faster than any invention in human history - with the possible exceptions of handguns and tequila." -Mitch Ratliffe
"Never trust a computer you can't throw out a window." -Steve Wozniak
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/sysl> og-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Aaahhhh. The web site was rebuilt and all the reference pages are now not accessible. Any links to the new location available? Richard
On Mon, Feb 17, 2003 at 09:36:08PM -0800, Richard E. Perlotto II wrote:
Aaahhhh. The web site was rebuilt and all the reference pages are now not accessible. Any links to the new location available?
we are still working on it. meanwhile you can find the reference pages in your syslog-ng tarball in the file named /doc/sgml/syslog-ng.html.tar.gz -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
On Tue, Feb 18, 2003 at 11:54:55AM +0100, Balazs Scheidler wrote:
On Mon, Feb 17, 2003 at 09:36:08PM -0800, Richard E. Perlotto II wrote:
Aaahhhh. The web site was rebuilt and all the reference pages are now not accessible. Any links to the new location available?
we are still working on it. meanwhile you can find the reference pages in your syslog-ng tarball in the file named /doc/sgml/syslog-ng.html.tar.gz
syslog-ng docs are now available at http://www.balabit.com/products/syslog_ng/reference/book1.html -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
participants (5)
-
Balazs Scheidler
-
Carson Gaspar
-
Nate Campi
-
Peter J. Holzer
-
Richard E. Perlotto II