[syslog-ng] [Bug 258] New: syslog-ng 3.4.2 stalls on Ubuntu 12.04 Precise with /var/ log full

[bugzilla at bugzilla.balabit.com]

https://bugzilla.balabit.com/show_bug.cgi?id=258

           Summary: syslog-ng 3.4.2 stalls on Ubuntu 12.04 Precise  with
                    /var/log full
           Product: syslog-ng
           Version: 3.4.x
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: unspecified
         Component: syslog-ng
        AssignedTo: bazsi at balabit.hu
        ReportedBy: carl.chenet-ext at cloudwatt.com
Type of the Report: bug
   Estimated Hours: 0.0


Hi,

We had several cases of frozen KVM virtual machines running Ubuntu 12.04 Precise with syslog-ng 3.4.2 taken from madhouse Debian repository.

Issue: the servers are not reachable any more. You can initiate a ssh/console connection but the connection freezes before displaying the banner.
Each time the /var/log lvm file system is full. The following errors occur in /var/log/system.log :

Oct 30 07:05:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:06:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:07:18 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current
message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\''
Oct 30 07:08:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:09:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:10:18 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current
message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\''
Oct 30 07:11:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:12:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:13:19 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current
message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\''
Oct 30 07:14:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:15:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
Oct 30 07:16:19 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current
message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\''
Oct 30 07:17:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)'

Workaround: We have the mcollective client installed on each server so we were able to kill the syslog-ng process by mcollective. when syslog-ng is killed,
the system stops freezing immedialty, the ssh connection is back.

I'm not sure to understand how syslog-ng freezes the system but it is worth reporting I guess.

Regards,
Carl Chenet


-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.