Thank you for quick answer, but unfortunately restarting syslog-ng is not an option.

Date: Thu, 21 Feb 2008 09:44:49 +0000 From: ac56@soas.ac.uk To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] syslog-ng does not resume log when disk is full at

Andrej Lobovski <drejc83@hotmail.com> [20080221 10:23:41 +0100]:

...

Hi.

I have a problem, with syslog-ng 2.0.0-1etch1 on debian 4. It is like this:

...eugh, maybe time to update. A large number of bugfixes have been made since then; particular the nasty memory leak issue[1]. Moving onto to 2.0.8 is dead easy with Debian, download the source and type 'debuild'; you then get yourself a package in the parent directory.

I'll try this.

...

- syslog-ng fills /var/log/syslog with messages - then /var partition becomes full and I get "No space left on device (28)" error written in /var/log/syslog (tail -f on this file), which is expected - then I make some disk space available on /var partition, BUT syslog-ng does not resume writing in /var/log/syslog at all

Am I missing any options in /etc/syslog-ng/syslog-ng.conf ?

When a filesystem fills up it's pretty normal to expect the worst and you should restart syslog-ng. My personal opinion is that if a disk full error fires up then an application should not be trying to add to the problem; how's that application to know 'enough' disk space has been freed up?

Well the same way that it produces a "no space left on device error (28)" and then retrying, let's say every 60 seconds, untill this error is gone. Can someone point me to this line or chunk of code, where I could add this? Ofcourse when I'm done with this error handling I will release my code for everyone to use. Thank you! Andrej _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

is dead easy with Debian, download the source and type 'debuild';

I can't seem to figure this out. EDIT: SOLVED forgot devscripts...shame on me :) _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Date: Thu, 21 Feb 2008 13:21:40 +0000 From: ac56@soas.ac.uk To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] syslog-ng does not resume log when disk is full at

Hi,

Andrej Lobovski [20080221 13:57:41 +0100]:

...

...

is dead easy with Debian, download the source and type 'debuild';

I can't seem to figure this out.

# aptitude install devscripts fakeroot $ mkdir -p /usr/src/deb-src/syslog-ng $ cd /usr/src/deb-src/syslog-ng $ wget http://www.balabit.com/downloads/files/syslog-ng/sources/stable/src/syslog-n... $ tar -zxf syslog-ng-2.0.8.tar.gz $ cd syslog-ng-2.0.8 $ debuild # dpkg -i ../syslog-ng_2.0.8_i386.deb

done.

Cheers

Thanks. I still have to find/make solution for the problem as I mustn't kill syslog-ng. Any help will do :). Cheers! _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Hi, Andrej Lobovski <drejc83@hotmail.com> [20080221 15:59:45 +0100]:

[snipped howto compile .deb for Debian]

I still have to find/make solution for the problem as I mustn't kill syslog-ng.

Any help will do :).

Sorry to be horribly brutal but surely time would be better spent on fixing the problem rather than helping at the recovery phase. The fact that syslog-ng is not able to dump data is a non-issue (the way I see it) when free space is created...the issue is that you have run out of space in the first place. The damage is *not* that syslog-ng will not gracefully recover, the real problem is that you just lost the previous nights logs as someone cunningly DoS'ed your syslog-ng server to fill up it's harddisk before they commenced an attack on your network! You solution: 1. 'rotate' your logs and compress and delete as need be. I have attached what I do below 2. spend the five minutes it would take you to write a script that monitors disk utilisation and lvextend/xfs_growfs as need be. Get it to automatically email you everytime it does this too The solution is trivial, waiting for Balazs to fix a bug where fundementally the issue is with your administrator is not the Right Way(tm) to deal with it. Sorry to sound harsh, but to me this is a no-brainer Cheers Alex ============ logging0:/home/ac56# cat /etc/cron.d/logging-remote-chores # m h dom mon dow user command # delete old logs, we keep for half a year 15 2 * * * root cd /var/log/remote/ && find . -type f -mtime +183 -exec rm '{}' \; # bzip up any logs more than a day old 15 3 * * * root cd /var/log/remote/ && find . -type f -mtime +0 ! -name '*.bz2' -exec nice -n 10 bzip2 -9 '{}' \; ============

Thank you for your input, but

problem is that you just lost the previous nights logs

this is actually not required in my case. I can actually lose all logs from time when disk was full, to the time when some disk space was freed. In my case it would be ok, that syslog-ng, when it runs out of disk space, writes everything from that point till the point it has some disk space, in dev/null instead in /var/logs/messages. Do you maybe have any clue on how to do that (as I am still lost in syslog-ng source code :) )? Thanks, Andrej _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Then you have two major problems: 1. why is all the date prior to 'disk full' getting lost

Sorry, i meant that logs that are lost because of disk full are not important. The logs up to the disk full are important, and also logs after disk is free are important. But I must not restart syslog-ng. so it goes like this boot linux, syslog-ng online, ... logging -> disk full ... not logging -> disk free ... logging again (does not matter if in same file than before disk was full or not whatever's easier :) ). syslog-ng must run all the time. cheers! Andrej _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Alexander Clouter <ac56@soas.ac.uk> [20080221 17:22:43 +0000]:

/me gives up with the final words of wisdom

"dude you data is toasted, you are preserving nothing by using the 0.5 seconds it takes to restart syslog-ng...just do it"

/me curse is mistyping... "dude your data is toast, you are loosing nothing by restarting syslog-ng which takes about 0.5 seconds....just simply do it and be done with it". Cheers Alex