syslog-ng uses 100% cpu when i use program() destination driver
Hi all, I have installed syslog-ng via apt-get on an Ubuntu 8.10 server. I have added the below to the configuration file: template t_smtp { template_escape(no); template("NOOP HELO localhost MAIL From: admin@example.com RCPT To: scott@example.com DATA From: admin@example.com To: scott@example.com Subject: Syslog msg from $HOST - [$FACILITY:$PRIORITY] $PROGRAM $S_DATE [$FACILITY:$PRIORITY] $MSG . QUIT "); }; log { source(s_all); filter(f_at_least_notice); destination(d_logfile); }; destination d_logfile { program("sendmail -bs > /var/log/syslog-email.log 2>&1" template(t_smtp)); }; For some reason with the above config after a message hits d_logfile the CPU usage jumps to 100% and stays there until syslog-ng is stopped. It's like some file it uses during the process gets locked and the thing jams up waiting forever. The only way to solve the problem is remove the program() entry and replace it with file("/var/log/temp.log" template(t_smtp)); in which case it works fine. To test the program()destination I created a php script that just slept for 10 seconds <?php sleep(10); ?> And called it in d_logfile { } with program("/usr/bin/php /tmp/sleep-test.php") And sure enough the cpu jumped to 100% and stayed there until syslog-ng was stopped. Can anyone see a program with my above config or suggest why syslog-ng can't cope when the program() doesn't return immediately? Thanks Scott ********************************************************************************** Disclaimer The information and attached documentation in this e-mail is intended for the use of the addressee only and is confidential. If you are not the intended recipient please delete it and notify us immediately by telephoning or e-mailing the sender. Please note that without Codemasters’ prior written consent any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Attachments to this e-mail may contain software viruses. You are advised to take all reasonable precautions to minimise this risk and to carry out a virus check on any documents before they are opened. Any offer contained in this communication is subject to Codemasters’ standard terms & conditions and must be signed by both parties. Except as expressly provided otherwise all information and attached documentation in this e-mail is subject to contract and Codemasters’ board approval. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Codemasters. This footnote also confirms that this email message has been swept by SurfControl for the presence of computer viruses. **********************************************************************************
Hi Scott, 100% cpu usage for an indefinite period after some event is the stereotypical symptom of an infinite loop in the event handler. And I can see exactly the cause of this loop in your "program" statement - it's a classic error that has been discussed on this list many times before. You should find good information about it in the archives - look for discussions of the "program" statement. In a nutshell: A syslog-ng "program" script is not permitted to exit after each message, it is required to read a series of individual messsages from its stdin and act on each one separately. If you write a script that exits after processing one message, it will cause exactly the sort of infinite loop you describe here. What's happening is that your script exits, syslog-ng starts it again, it exits again, syslog-ng starts it again, it exits again, syslog-ng starts it again, it exits again, syslog-ng starts it again, and on and on and on. See the syslog-ng docs and previous posts to this list for information on how the script behind a "program" destination is required to work. Joe. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scott McGillivray Sent: 10 December 2008 10:48 To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] syslog-ng uses 100% cpu when i use program() destination driver Hi all, I have installed syslog-ng via apt-get on an Ubuntu 8.10 server. I have added the below to the configuration file: template t_smtp { template_escape(no); template("NOOP HELO localhost MAIL From: admin@example.com RCPT To: scott@example.com DATA From: admin@example.com To: scott@example.com Subject: Syslog msg from $HOST - [$FACILITY:$PRIORITY] $PROGRAM $S_DATE [$FACILITY:$PRIORITY] $MSG . QUIT "); }; log { source(s_all); filter(f_at_least_notice); destination(d_logfile); }; destination d_logfile { program("sendmail -bs > /var/log/syslog-email.log 2>&1" template(t_smtp)); }; For some reason with the above config after a message hits d_logfile the CPU usage jumps to 100% and stays there until syslog-ng is stopped. It's like some file it uses during the process gets locked and the thing jams up waiting forever. The only way to solve the problem is remove the program() entry and replace it with file("/var/log/temp.log" template(t_smtp)); in which case it works fine. To test the program()destination I created a php script that just slept for 10 seconds <?php sleep(10); ?> And called it in d_logfile { } with program("/usr/bin/php /tmp/sleep-test.php") And sure enough the cpu jumped to 100% and stayed there until syslog-ng was stopped. Can anyone see a program with my above config or suggest why syslog-ng can't cope when the program() doesn't return immediately? Thanks Scott ********************************************************************************** Disclaimer The information and attached documentation in this e-mail is intended for the use of the addressee only and is confidential. If you are not the intended recipient please delete it and notify us immediately by telephoning or e-mailing the sender. Please note that without Codemasters' prior written consent any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Attachments to this e-mail may contain software viruses. You are advised to take all reasonable precautions to minimise this risk and to carry out a virus check on any documents before they are opened. Any offer contained in this communication is subject to Codemasters' standard terms & conditions and must be signed by both parties. Except as expressly provided otherwise all information and attached documentation in this e-mail is subject to contract and Codemasters' board approval. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Codemasters. This footnote also confirms that this email message has been swept by SurfControl for the presence of computer viruses. ********************************************************************************** ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Many thanks Joe. I found the posts you referred to and amended the config by creating a bash file with #!/bin/bash while read LINE; do sendmail -bs > /var/log/syslog-nc-email.log 2>&1 $LINE Done And changed the program() to use the script and existing template program("/tmp/send-syslogmail.sh" template(t_smtp)); Seems to work fine with no cpu 100% issues and for my single server low volume needs it should cope quite nicely. Thanks again Scott -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Fegan, Joe Sent: 10 December 2008 11:24 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng uses 100% cpu when i useprogram() destination driver Hi Scott, 100% cpu usage for an indefinite period after some event is the stereotypical symptom of an infinite loop in the event handler. And I can see exactly the cause of this loop in your "program" statement - it's a classic error that has been discussed on this list many times before. You should find good information about it in the archives - look for discussions of the "program" statement. In a nutshell: A syslog-ng "program" script is not permitted to exit after each message, it is required to read a series of individual messsages from its stdin and act on each one separately. If you write a script that exits after processing one message, it will cause exactly the sort of infinite loop you describe here. What's happening is that your script exits, syslog-ng starts it again, it exits again, syslog-ng starts it again, it exits again, syslog-ng starts it again, it exits again, syslog-ng starts it again, and on and on and on. See the syslog-ng docs and previous posts to this list for information on how the script behind a "program" destination is required to work. Joe. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scott McGillivray Sent: 10 December 2008 10:48 To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] syslog-ng uses 100% cpu when i use program() destination driver Hi all, I have installed syslog-ng via apt-get on an Ubuntu 8.10 server. I have added the below to the configuration file: template t_smtp { template_escape(no); template("NOOP HELO localhost MAIL From: admin@example.com RCPT To: scott@example.com DATA From: admin@example.com To: scott@example.com Subject: Syslog msg from $HOST - [$FACILITY:$PRIORITY] $PROGRAM $S_DATE [$FACILITY:$PRIORITY] $MSG . QUIT "); }; log { source(s_all); filter(f_at_least_notice); destination(d_logfile); }; destination d_logfile { program("sendmail -bs > /var/log/syslog-email.log 2>&1" template(t_smtp)); }; For some reason with the above config after a message hits d_logfile the CPU usage jumps to 100% and stays there until syslog-ng is stopped. It's like some file it uses during the process gets locked and the thing jams up waiting forever. The only way to solve the problem is remove the program() entry and replace it with file("/var/log/temp.log" template(t_smtp)); in which case it works fine. To test the program()destination I created a php script that just slept for 10 seconds <?php sleep(10); ?> And called it in d_logfile { } with program("/usr/bin/php /tmp/sleep-test.php") And sure enough the cpu jumped to 100% and stayed there until syslog-ng was stopped. Can anyone see a program with my above config or suggest why syslog-ng can't cope when the program() doesn't return immediately? Thanks Scott ************************************************************************ ********** Disclaimer The information and attached documentation in this e-mail is intended for the use of the addressee only and is confidential. If you are not the intended recipient please delete it and notify us immediately by telephoning or e-mailing the sender. Please note that without Codemasters' prior written consent any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Attachments to this e-mail may contain software viruses. You are advised to take all reasonable precautions to minimise this risk and to carry out a virus check on any documents before they are opened. Any offer contained in this communication is subject to Codemasters' standard terms & conditions and must be signed by both parties. Except as expressly provided otherwise all information and attached documentation in this e-mail is subject to contract and Codemasters' board approval. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Codemasters. This footnote also confirms that this email message has been swept by SurfControl for the presence of computer viruses. ************************************************************************ ********** ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ********************************************************************************** Disclaimer The information and attached documentation in this e-mail is intended for the use of the addressee only and is confidential. If you are not the intended recipient please delete it and notify us immediately by telephoning or e-mailing the sender. Please note that without Codemasters’ prior written consent any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Attachments to this e-mail may contain software viruses. You are advised to take all reasonable precautions to minimise this risk and to carry out a virus check on any documents before they are opened. Any offer contained in this communication is subject to Codemasters’ standard terms & conditions and must be signed by both parties. Except as expressly provided otherwise all information and attached documentation in this e-mail is subject to contract and Codemasters’ board approval. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Codemasters. This footnote also confirms that this email message has been swept by SurfControl for the presence of computer viruses. **********************************************************************************
participants (2)
-
Fegan, Joe
-
Scott McGillivray