It's not silly and in fact I've set flush_lines (10); in syslog-ng.conf. I know it's not a lot, but 10 lines is far more than crond outputs per minute, which is why I don't understand syslog-ng writes every minute when crond is running and almost not at all when crond is not running. Am I making any sense? :) Lars On Mon, Jul 11, 2011 at 7:10 PM, Paul Krizak <paul.krizak@amd.com> wrote:

This may be a silly question, but have you tried setting flush_timeout() to some large value so that messages are queued?  Or is this not the behavior you seek?  It seems fairly obvious to me that if you have anything writing to syslog at any time, and those messages end  up in a file, that the kernel is going to spin up the disks and write out the dirty pages.

The only way I can imagine to avoid this would be to queue the messages within syslog-ng (which won't cause any disk-backed pages in memory to become dirty) and only write them out at wide enough intervals that the kernel sees enough inactivity to spin down the disks.

Paul Krizak                         7171 Southwest Pkwy MS B200.3A MTS Systems Engineer                Austin, TX  78735 Advanced Micro Devices              Desk:  (512) 602-8775 Linux/Unix Systems Engineering      Cell:  (512) 791-0686 Global IT Infrastructure            Fax:   (512) 602-0468

On 07/11/2011 09:58 AM, Lars Stokholm wrote:

...

Hello

On my laptop I run Arch Linux with syslog-ng and cronie (cron) among other programs of course.

I'm trying to get my harddrive to spin down automatically once in a while (using hdparm).

Me and a lot of other people both in the Arch and Ubuntu community are having our attempts to do this ruined, by what first seemed to be the EXT4 journalling block device (jbd2).

After long discussions and finally a bug report (see at the bottom of this mail), I think I've boiled my troubles down to be mostly about syslog-ng (and possibly cronie - but I don't think so).

No matter what I try I can't seem to get syslog-ng to stop "waking up my disks" (at least that's what kernel tracing tells me). Killing the crond process almost totally stops these syslog-ng writes to the disk. Unfortunately I've been unable to configure my way out of it using syslog-ng.conf. So my harddrive won't stay spinned down - not even for a minute.

This is what tracing the journalling gives me (see the kernel bug report below for an explanaition): http://pastebin.com/raw.php?i=J6BtGVrz

As you can probably see, syslog-ng writes every minute at least. I guess that's because my user crontab has a command that gets run every minute. What I don't get is why syslog-ng writes to the disk, even though I'm telling it not to log crond's output.

Here is my syslog-ng.conf - as you can see I tried to output the crond messages to /dev/null. I guess it works, because there are no mentions of crond anywhere in /var/log anymore. http://pastebin.com/raw.php?i=H02KKWPV

Here are the discussions and the bug report I mentioned: https://bbs.archlinux.org/viewtopic.php?id=113516 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/607560 https://bugzilla.kernel.org/show_bug.cgi?id=39072

I don't know if syslog-ng is causing trouble for only me, but it seems unlikely.

Regards, Lars

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Oh, I thought you wrote flush_lines. I don't have a flush_timeout in my syslog-ng.conf. I see you set it in milliseconds. But what is the default and what value would you recommend? Lars On Mon, Jul 11, 2011 at 7:21 PM, Paul Krizak <paul.krizak@amd.com> wrote:

flush_timeout() and flush_lines() work together.  Syslog-ng will write to disk if flush_lines() lines are in the buffer, *or* if flush_timeout() happens, whichever happens first.

So to get your desired behavior you should set flush_timeout() to 60 seconds or something, that way the 2-3 lines in the buffer will stay there long enough for the kernel to spin down the disks.

Paul Krizak                         7171 Southwest Pkwy MS B200.3A MTS Systems Engineer                Austin, TX  78735 Advanced Micro Devices              Desk:  (512) 602-8775 Linux/Unix Systems Engineering      Cell:  (512) 791-0686 Global IT Infrastructure            Fax:   (512) 602-0468

On 07/11/2011 10:19 AM, Lars Stokholm wrote:

...

It's not silly and in fact I've set   flush_lines (10);   in syslog-ng.conf. I know it's not a lot, but 10 lines is far more than crond outputs per minute, which is why I don't understand syslog-ng writes every minute when crond is running and almost not at all when crond is not running. Am I making any sense? :)

Lars

On Mon, Jul 11, 2011 at 7:10 PM, Paul Krizak<paul.krizak@amd.com>  wrote:

...

This may be a silly question, but have you tried setting flush_timeout() to some large value so that messages are queued?  Or is this not the behavior you seek?  It seems fairly obvious to me that if you have anything writing to syslog at any time, and those messages end  up in a file, that the kernel is going to spin up the disks and write out the dirty pages.

The only way I can imagine to avoid this would be to queue the messages within syslog-ng (which won't cause any disk-backed pages in memory to become dirty) and only write them out at wide enough intervals that the kernel sees enough inactivity to spin down the disks.

Paul Krizak                         7171 Southwest Pkwy MS B200.3A MTS Systems Engineer                Austin, TX  78735 Advanced Micro Devices              Desk:  (512) 602-8775 Linux/Unix Systems Engineering      Cell:  (512) 791-0686 Global IT Infrastructure            Fax:   (512) 602-0468

On 07/11/2011 09:58 AM, Lars Stokholm wrote:

...

Hello

On my laptop I run Arch Linux with syslog-ng and cronie (cron) among other programs of course.

I'm trying to get my harddrive to spin down automatically once in a while (using hdparm).

Me and a lot of other people both in the Arch and Ubuntu community are having our attempts to do this ruined, by what first seemed to be the EXT4 journalling block device (jbd2).

After long discussions and finally a bug report (see at the bottom of this mail), I think I've boiled my troubles down to be mostly about syslog-ng (and possibly cronie - but I don't think so).

No matter what I try I can't seem to get syslog-ng to stop "waking up my disks" (at least that's what kernel tracing tells me). Killing the crond process almost totally stops these syslog-ng writes to the disk. Unfortunately I've been unable to configure my way out of it using syslog-ng.conf. So my harddrive won't stay spinned down - not even for a minute.

This is what tracing the journalling gives me (see the kernel bug report below for an explanaition): http://pastebin.com/raw.php?i=J6BtGVrz

As you can probably see, syslog-ng writes every minute at least. I guess that's because my user crontab has a command that gets run every minute. What I don't get is why syslog-ng writes to the disk, even though I'm telling it not to log crond's output.

Here is my syslog-ng.conf - as you can see I tried to output the crond messages to /dev/null. I guess it works, because there are no mentions of crond anywhere in /var/log anymore. http://pastebin.com/raw.php?i=H02KKWPV

Here are the discussions and the bug report I mentioned: https://bbs.archlinux.org/viewtopic.php?id=113516 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/607560 https://bugzilla.kernel.org/show_bug.cgi?id=39072

I don't know if syslog-ng is causing trouble for only me, but it seems unlikely.

Regards, Lars

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

I see, that makes sense. Thanks. I would also like to avoid it if possible. But in order to avoid it, I have to find out why this is happening:

...

...

...

...

...

As you can probably see, syslog-ng writes every minute at least. I guess that's because my user crontab has a command that gets run every minute. What I don't get is why syslog-ng writes to the disk, even though I'm telling it not to log crond's output.

Any ideas? As you can see from my syslog-ng.conf, everything that has to do with crond is sent to /dev/null. So crond output shouldn't cause a spin-up of my harddrive, but it does. Killing crond or removing the once-per-minute-command from my user crontab will allow my harddrive to stay spinned down for a long period of time. But I would rather not do either.

Are you doing anything with syslog-ng's internal log messages? It's maintaining counters and such internally that I believe get written out to disk by default once per minute. But since that's a normal log path, that should be controlled by flush_timeout() as well. Start by trying the flush_timeout() setting, and if that works, then at least you know that is something in a log path somewhere causing the trouble. Another thing you could try is removing all of your log paths (not just routing them to /dev/null) and see how syslog-ng behaves. Finally, if you still can't nail it down, use strace to trace syslog-ng during a period where it causes the disks to spin up. You should be able to at least see the file it's trying to write. For example, it might be updating its pid file in /var/run once per minute. Paul Krizak 7171 Southwest Pkwy MS B200.3A MTS Systems Engineer Austin, TX 78735 Advanced Micro Devices Desk: (512) 602-8775 Linux/Unix Systems Engineering Cell: (512) 791-0686 Global IT Infrastructure Fax: (512) 602-0468 On 07/11/2011 11:33 AM, Lars Stokholm wrote:

And I forgot to add to that, that killing syslog-ng, but keeping crond alive, will also allow my harddrive to stay spinned down, so this is not crond's fault. As soon as I start syslog-ng I see log entries in trace_pipe and my disk starts to wake up a lot.

On Mon, Jul 11, 2011 at 8:28 PM, Lars Stokholm<lars.stokholm@gmail.com> wrote:

...

I see, that makes sense. Thanks. I would also like to avoid it if possible. But in order to avoid it, I have to find out why this is happening:

...

...

...

...

>> As you can probably see, syslog-ng writes every minute at least. I guess >> that's because my user crontab has a command that gets run every minute. >> What I don't get is why syslog-ng writes to the disk, even though I'm >> telling it not to log crond's output.

Any ideas? As you can see from my syslog-ng.conf, everything that has to do with crond is sent to /dev/null. So crond output shouldn't cause a spin-up of my harddrive, but it does. Killing crond or removing the once-per-minute-command from my user crontab will allow my harddrive to stay spinned down for a long period of time. But I would rather not do either.

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

syslog-ng-bounces@lists.balabit.hu wrote on 12/07/2011 06:02:57:

I did, on the other hand, try running strace on syslog-ng and this is was I got just before my harddrive would wake up, every time: http://pastebin.com/raw.php?i=hFZVurBP

What can you make from this strace output? The same pattern was repeated again and again. I guess "/usr/sbin/cr" is the cut-off path to crond.

Try an 'lsof -p <syslog-ng pid>' to identify what file descriptors 9, 10 and 11 are; that will tell you what files it is polling (or, judging by the write()s, updating) on each occasion. It would appear you haven't successfully disabled cron's logging in your syslog-ng configuration. Regards, Scott Rochford

I have a guarantee that it'll spin up every 10 minutes, forever, although there's absolutely no reason for it.

There absolutely is a reason for your disks to spin up -- applications are sending messages to syslog-ng, and syslog-ng is dutifully trying to write them to disk. If you aren't writing it out to disk, then what's the point of running syslog at all? If you really don't care about keeping the contents of your system logs (and you should), you could always create a ramdisk in your startup sequence (before syslog-ng starts up) and then mount /var/log to it. Then writes to syslog wouldn't hit the hard disk, and your HDD can stay spun down. Since you probably wouldn't want to waste much RAM on that, you should probably also combine that with some modified logrotate configuration and run logrotate hourly instead of daily, to ensure you don't fill up your ramdisk with log data.

I did, on the other hand, try running strace on syslog-ng and this is was I got just before my harddrive would wake up, every time: http://pastebin.com/raw.php?i=hFZVurBP

If you run strace with -s 1024 you'd be able to see the whole message that is being written, but it looks like a cron message to me, which makes sense. Cron wakes up regularly, and when it does, it sends a syslog message. syslog-ng receives that message, and in the default configuration, will promptly commit that message to the log on disk. Honestly I'm not really sure what you're so concerned about here -- your computer is always active doing *something*, even when you're not actually sitting in front of it. Part of that is the system logging activity, which has to be able to write to disk. If you're trying to eke out every last watt from your laptop to enhance battery life, then I would think that using an low-power SSD instead of a spinning disk would be a better trade than trying to make your computer completely avoid making disk accesses when idle. Paul Krizak 7171 Southwest Pkwy MS B200.3A MTS Systems Engineer Austin, TX 78735 Advanced Micro Devices Desk: (512) 602-8775 Linux/Unix Systems Engineering Cell: (512) 791-0686 Global IT Infrastructure Fax: (512) 602-0468 On 07/11/2011 01:02 PM, Lars Stokholm wrote:

What do you mean by it's internal log messages? These? destination d_syslog { file("/var/log/syslog.log"); };

flush_timeout worked perfectly. I tried to set it to 10 minutes and that's exactly how long my harddrive was allowed to stay spun down. But on the other hand I have a guarantee that it'll spin up every 10 minutes, forever, although there's absolutely no reason for it.

I didn't try removing all the log paths, since I was not sure exactly how I was supposed to do it and since it's not really clear to me what I'll gain from it. :)

I did, on the other hand, try running strace on syslog-ng and this is was I got just before my harddrive would wake up, every time: http://pastebin.com/raw.php?i=hFZVurBP

What can you make from this strace output? The same pattern was repeated again and again. I guess "/usr/sbin/cr" is the cut-off path to crond.

On Mon, Jul 11, 2011 at 8:49 PM, Paul Krizak<paul.krizak@amd.com> wrote:

...

Are you doing anything with syslog-ng's internal log messages? It's maintaining counters and such internally that I believe get written out to disk by default once per minute.

But since that's a normal log path, that should be controlled by flush_timeout() as well.

Start by trying the flush_timeout() setting, and if that works, then at least you know that is something in a log path somewhere causing the trouble.

Another thing you could try is removing all of your log paths (not just routing them to /dev/null) and see how syslog-ng behaves.

Finally, if you still can't nail it down, use strace to trace syslog-ng during a period where it causes the disks to spin up. You should be able to at least see the file it's trying to write. For example, it might be updating its pid file in /var/run once per minute.

Paul Krizak 7171 Southwest Pkwy MS B200.3A MTS Systems Engineer Austin, TX 78735 Advanced Micro Devices Desk: (512) 602-8775 Linux/Unix Systems Engineering Cell: (512) 791-0686 Global IT Infrastructure Fax: (512) 602-0468

On 07/11/2011 11:33 AM, Lars Stokholm wrote:

...

And I forgot to add to that, that killing syslog-ng, but keeping crond alive, will also allow my harddrive to stay spinned down, so this is not crond's fault. As soon as I start syslog-ng I see log entries in trace_pipe and my disk starts to wake up a lot.

On Mon, Jul 11, 2011 at 8:28 PM, Lars Stokholm<lars.stokholm@gmail.com> wrote:

...

I see, that makes sense. Thanks. I would also like to avoid it if possible. But in order to avoid it, I have to find out why this is happening:

...

...

>>>> As you can probably see, syslog-ng writes every minute at least. I guess >>>> that's because my user crontab has a command that gets run every minute. >>>> What I don't get is why syslog-ng writes to the disk, even though I'm >>>> telling it not to log crond's output.

Any ideas? As you can see from my syslog-ng.conf, everything that has to do with crond is sent to /dev/null. So crond output shouldn't cause a spin-up of my harddrive, but it does. Killing crond or removing the once-per-minute-command from my user crontab will allow my harddrive to stay spinned down for a long period of time. But I would rather not do either.

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq