log_fifo_size and locked tables
'allo all, I've 1.6.9 in a centralized environment logging to MySQL. I'm currently working up a rotate system whereby: - the db is locked - mysqldump, gzip, scp off the box - rows with a timestamp older than 7 days are deleted - unlock - carry on. and was wondering a couple things. Firstly, 1) is this a horribly bad idea that should be replaced with a completely different plan? failing that, 2) How long does the default log_fifo_size of 2048 (lines, yes?) hold up, volume-wise? That is, while the tables are locked, i am assuming that this is where messages start piling up until they're unlocked. At the moment, i'm not dealing with high volume, but once everything seems in place i'm going to be adding many more hosts. is there any kind of rule of thumb for this value vis-a-vis the logs generated? At what sort of daily volume should i look towards upping it? Thanks! -- Rob Munsch Solutions For Progress IT
Hi, Rob Munsch <rmunsch@solutionsforprogress.com> [20060411 15:55:20 -0400]:
'allo all,
I've 1.6.9 in a centralized environment logging to MySQL. I'm currently working up a rotate system whereby: - the db is locked - mysqldump, gzip, scp off the box - rows with a timestamp older than 7 days are deleted - unlock - carry on.
I would suggest a slight order alteration, the key is to always lock the database for as short a time as possible: 1. dump - locks for you and then unlocks [2. lock?] 3. purge 'junk' [4. unlock?] 5. gzip, scp whatever The 'carry on' bit would happen whilst the gzip is taking place in the background, just make sure you nice up the process to something like 19 or something so the computer will only use *spare* CPU cycles to gzip/scp your mysqldump data about. Just as a side note, if you simply purging rows with a simple SQL statement I'm pretty sure you can drop the locking/unlocking step altogether; its only if you need to maintain 'state' where objects are related to other objects. In your case you are simply chopping off the bottom of the table. This probably does not hold too well in an MySQL cluster I'm guessing....but thats starting to get way above my head there :) An optimisation would be to index the 'date' column which only contains 'year-month-day' so that instantly the SQL database can scrub those entries, your 'lock' would really not be in place then for any length of time.
and was wondering a couple things. Firstly, 1) is this a horribly bad idea that should be replaced with a completely different plan? failing that,
Well it looks like you always want at least seven days of data in the database so its hard to think of an alternative method.
2) How long does the default log_fifo_size of 2048 (lines, yes?) hold up, volume-wise? That is, while the tables are locked, i am assuming that this is where messages start piling up until they're unlocked. At the moment, i'm not dealing with high volume, but once everything seems in place i'm going to be adding many more hosts.
If you do lock/unlock you might want to break up things into smaller chunks to give the chance for the buffers to flush, so: 1 lock 2. purge day 14 3. unlock 4. lock 5. purge day 13 6. unlock 7. lock 8. purge day 12 ....etc etc to 30ish. lock 31. purge day 7 32. unlock Inbetween the locks/unlocks any buffers could be processed....however to be honest if you index your 'date' column the above will probably be a terrible and pointless approach.
is there any kind of rule of thumb for this value vis-a-vis the logs generated? At what sort of daily volume should i look towards upping it?
Thats for the others here to deal with.... Cheers Alex
Thanks!
-- Rob Munsch Solutions For Progress IT
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Thank you! This is a lot of good information. Alexander Clouter wrote:
Well it looks like you always want at least seven days of data in the database so its hard to think of an alternative method.
Seven is a number i pulled out of my - um, hat. I'm wanting to maximize the usefulness of the 'live' database of logs (using phpmysyslogng atm), but i suppose seven is kind of high, isn't it? I just want the searchable-via-frontend data to be able to provide some history and context if we go looking for something. What would you suggest? -- Rob Munsch Solutions For Progress IT
Hi, Rob Munsch <rmunsch@solutionsforprogress.com> [20060412 12:22:02 -0400]:
Thank you! This is a lot of good information.
Alexander Clouter wrote:
Well it looks like you always want at least seven days of data in the database so its hard to think of an alternative method.
Seven is a number i pulled out of my - um, hat. I'm wanting to maximize the usefulness of the 'live' database of logs (using phpmysyslogng atm), but i suppose seven is kind of high, isn't it? I just want the searchable-via-frontend data to be able to provide some history and context if we go looking for something.
What would you suggest?
To be honest, if you are indexing on the date then you could probably get away with as much recent historical data as you want. Start at 14 days or something but if you find it overwhelming or not needed trim it. As you are archiving all the old data you could, if you needed older data, is import it back in and at the next weeks clearout of old logfiles they will be automagically purged. I think you should be safe with as much as you want, as long as you index :) Cheers Alex
List, I have syslog-ng 1.6.x running successfully on several Panther servers, but I cannot for the life of me get 1.6.x or 1.9.x to log ANYTHING on my newly upgraded Tiger server, leaving me stuck with the stock syslogd. I¹ve tried messing with the config and installing on a 10.4.x client machine with no luck as well. I¹ve even ktraced syslog-ng as it is launched but I¹ve seen nothing convicting. Have there been changes to the logging mechanisms in 10.4? What¹s the trick to getting this running on 10.4.x? Thanks, -- Alan Orth User Services - Field Services Technician Upward Bound - Systems Administrator California State University, Chico (530)898-6000
Also relevant, When I ktrace syslog-ng on either the server or the client Tiger install, the log is FILLED with polls... The kdump would fill my hard drive if I forgot to stop it, hah. And you can probably guess that CPU usage for syslog-ng is up around 70 80%... Thanks, -- Alan Orth aorth@csuchico.edu User Services - Field Services Technician Upward Bound - Systems Administrator California State University, Chico (530)898-6000 From: Alan Orth <aorth@exchange.csuchico.edu> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Date: Mon, 17 Apr 2006 00:35:20 -0700 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Conversation: Syslog-ng on Mac OS X Tiger? Subject: [syslog-ng] Syslog-ng on Mac OS X Tiger? List, I have syslog-ng 1.6.x running successfully on several Panther servers, but I cannot for the life of me get 1.6.x or 1.9.x to log ANYTHING on my newly upgraded Tiger server, leaving me stuck with the stock syslogd. I¹ve tried messing with the config and installing on a 10.4.x client machine with no luck as well. I¹ve even ktraced syslog-ng as it is launched but I¹ve seen nothing convicting. Have there been changes to the logging mechanisms in 10.4? What¹s the trick to getting this running on 10.4.x? Thanks, -- Alan Orth User Services - Field Services Technician Upward Bound - Systems Administrator California State University, Chico (530)898-6000 _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Apr 17, 2006, at 12:03 PM, Orth, Alan wrote:
Also relevant,
When I ktrace syslog-ng on either the server or the client Tiger install, the log is FILLED with polls... The kdump would fill my hard drive if I forgot to stop it, hah. And you can probably guess that CPU usage for syslog-ng is up around 70 – 80%...
Thanks,
List,
I have syslog-ng 1.6.x running successfully on several Panther servers, but I cannot for the life of me get 1.6.x or 1.9.x to log ANYTHING on my newly upgraded Tiger server, leaving me stuck with the stock syslogd.
I’ve tried messing with the config and installing on a 10.4.x client machine with no luck as well. I’ve even ktraced syslog-ng as it is launched but I’ve seen nothing convicting.
Have there been changes to the logging mechanisms in 10.4? What’s the trick to getting this running on 10.4.x?
Thanks,
-- Alan Orth
It needs to start up after the network has come up in Tiger, otherwise it blocks and never recovers. Darkshadow (aka Michael Nickerson)
Darkshadow, In Panther I wrote a StartupItem for syslog-ng that had it ³require² the Resolver, this worked fine for startup. The thing is, syslog-ng doesn¹t log at all in either of my Tiger installs. Ever. Even when I start it up manually. For example, launchctl unload /System/Library/LaunchDaemons/com.apple.syslod.plist syslog-ng -f /usr/local/etc/syslog-ng.conf And then it gets as far as creating that day¹s logfile... But it stays empty and then polls for ever... -- Alan Orth aorth@csuchico.edu User Services - Field Services Technician Upward Bound - Systems Administrator California State University, Chico (530)898-6000 From: Darkshadow <darkshadow02@mac.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Date: Mon, 17 Apr 2006 15:30:56 -0400 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Syslog-ng on Mac OS X Tiger? needs to start up after the network has come up in Tiger, otherwise it blocks and never recovers.
participants (5)
-
Alan Orth
-
Alexander Clouter
-
Darkshadow
-
Orth, Alan
-
Rob Munsch