program destination and init??
Hi, I've got a question regarding the program destination and how/when it spawns the program. I've got a few program destination directives in my syslog-ng config. When I start the syslog-ng daemon, the program's (of the program destinations) parents are the syslog-ng daemon. After some amount of time (perhaps 20 or 30 hours) I get more programs spawned. Some of the programs have parents that are 'init' (PID = 1). One of my programs is 'vpndb' - see the grep output for an example: $ ps -ef | grep vpndb root 610 1 41 Nov23 ? 2-01:29:55 perl /usr/bin/vpndb root 4036 605 0 06:44 ? 00:00:00 /bin/sh -c /usr/bin/vpndb root 4039 4036 0 06:44 ? 00:00:00 perl /usr/bin/vpndb root 5231 1 26 Nov24 ? 1-02:09:12 perl /usr/bin/vpndb root 5322 1 1 Nov27 ? 00:21:26 perl /usr/bin/vpndb root 5658 1381 0 08:33 pts/4 00:00:00 grep vpndb root 16167 1 12 Nov26 ? 06:19:10 perl /usr/bin/vpndb root 26542 1 19 Nov25 ? 14:12:19 perl /usr/bin/vpndb Above there are 5 vpndb programs with a parent of PID = 1. One of the vpndb's belongs to syslog-ng. Two of my other program destinations are "ssh" commands. They exhibit the same symptoms (multiple programs running with parents of PID = 1). Hence I don't think this problem is related to the specific program being executed. Does anyone have any idea why I am getting multiple programs spawned? (I presume that syslog-ng is spawning these programs.) I do have cron jobs that will 'reload' the syslog-ng daemon. Thanks for any hints. -matt zagrabelny
On Mon, Nov 28, 2011 at 9:00 AM, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Hi,
I've got a question regarding the program destination and how/when it spawns the program.
I've got a few program destination directives in my syslog-ng config. When I start the syslog-ng daemon, the program's (of the program destinations) parents are the syslog-ng daemon. After some amount of time (perhaps 20 or 30 hours) I get more programs spawned. Some of the programs have parents that are 'init' (PID = 1). One of my programs is 'vpndb' - see the grep output for an example:
$ ps -ef | grep vpndb root 610 1 41 Nov23 ? 2-01:29:55 perl /usr/bin/vpndb root 4036 605 0 06:44 ? 00:00:00 /bin/sh -c /usr/bin/vpndb root 4039 4036 0 06:44 ? 00:00:00 perl /usr/bin/vpndb root 5231 1 26 Nov24 ? 1-02:09:12 perl /usr/bin/vpndb root 5322 1 1 Nov27 ? 00:21:26 perl /usr/bin/vpndb root 5658 1381 0 08:33 pts/4 00:00:00 grep vpndb root 16167 1 12 Nov26 ? 06:19:10 perl /usr/bin/vpndb root 26542 1 19 Nov25 ? 14:12:19 perl /usr/bin/vpndb
Above there are 5 vpndb programs with a parent of PID = 1. One of the vpndb's belongs to syslog-ng.
Two of my other program destinations are "ssh" commands. They exhibit the same symptoms (multiple programs running with parents of PID = 1). Hence I don't think this problem is related to the specific program being executed.
Does anyone have any idea why I am getting multiple programs spawned? (I presume that syslog-ng is spawning these programs.)
I do have cron jobs that will 'reload' the syslog-ng daemon.
Replying to myself.
From empirical tests it does look like the 'reload' is causing the extra programs to get spawned.
I am using the syslog-ng from Debian Squeeze: $ dpkg -l syslog-ng syslog-ng 3.1.3-3 And reloading the daemon is done via start-stop-daemon: start-stop-daemon --stop --signal 1 --quiet --exec "$SYSLOGNG" \ --pidfile "$PIDFILE" which to me, there doesn't look to be anything that would cause the extra program spawning in the above command. Any ideas why reloading the daemon is causing extra programs to be "spawned"? Thanks for the help! -mz
Hi, I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected. hope it helps, regards, Tom Zitat von Matt Zagrabelny <mzagrabe@d.umn.edu>:
On Mon, Nov 28, 2011 at 9:00 AM, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Hi,
I've got a question regarding the program destination and how/when it spawns the program.
I've got a few program destination directives in my syslog-ng config. When I start the syslog-ng daemon, the program's (of the program destinations) parents are the syslog-ng daemon. After some amount of time (perhaps 20 or 30 hours) I get more programs spawned. Some of the programs have parents that are 'init' (PID = 1). One of my programs is 'vpndb' - see the grep output for an example:
$ ps -ef | grep vpndb root 610 1 41 Nov23 ? 2-01:29:55 perl /usr/bin/vpndb root 4036 605 0 06:44 ? 00:00:00 /bin/sh -c /usr/bin/vpndb root 4039 4036 0 06:44 ? 00:00:00 perl /usr/bin/vpndb root 5231 1 26 Nov24 ? 1-02:09:12 perl /usr/bin/vpndb root 5322 1 1 Nov27 ? 00:21:26 perl /usr/bin/vpndb root 5658 1381 0 08:33 pts/4 00:00:00 grep vpndb root 16167 1 12 Nov26 ? 06:19:10 perl /usr/bin/vpndb root 26542 1 19 Nov25 ? 14:12:19 perl /usr/bin/vpndb
Above there are 5 vpndb programs with a parent of PID = 1. One of the vpndb's belongs to syslog-ng.
Two of my other program destinations are "ssh" commands. They exhibit the same symptoms (multiple programs running with parents of PID = 1). Hence I don't think this problem is related to the specific program being executed.
Does anyone have any idea why I am getting multiple programs spawned? (I presume that syslog-ng is spawning these programs.)
I do have cron jobs that will 'reload' the syslog-ng daemon.
Replying to myself.
From empirical tests it does look like the 'reload' is causing the extra programs to get spawned.
I am using the syslog-ng from Debian Squeeze:
$ dpkg -l syslog-ng syslog-ng 3.1.3-3
And reloading the daemon is done via start-stop-daemon:
start-stop-daemon --stop --signal 1 --quiet --exec "$SYSLOGNG" \ --pidfile "$PIDFILE"
which to me, there doesn't look to be anything that would cause the extra program spawning in the above command.
Any ideas why reloading the daemon is causing extra programs to be "spawned"?
Thanks for the help!
-mz ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On Mon, Nov 28, 2011 at 11:53 AM, Thomas Wollner <tw@wollner-net.de> wrote:
Hi,
I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected.
hope it helps,
Tom, It does help. Thanks for your tip! -mz
Thomas Wollner <tw@wollner-net.de> writes:
I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected.
Ooh, this is interesting. Thanks for figuring this out, I'll see if I can figure out how to make it work with dash too. (Or if it's a bug in dash that should be fixed) -- |8]
Hi, currently I`m using syslog-ng 3.3.3 from madhouse package. In addition to the behaviour with the dash shell I found out that 3.3.3 does not handle signalling to scripts spawned by program destination. Example: in 3.1.3 this was a valid config statement: destination d_raw_firewall_sec { program("`/opt/sec/bin/secStart firewall`" template(t_sec_template)); }; Have a look at the backticks inside the program. After changing /bin/dash nach /bin/bash everything worked fine. in 3.3.3. I have to change from backtick to $(...) style: destination d_raw_firewall_sec { program("$(/opt/sec/bin/secStart firewall)" template(t_sec_template)); }; This works, but upon restart / reload of syslog-ng the "old" spawned scripts are not receiving the kill / term / hup signals and stay in the process list of the system until killed manually. The $(/opt/sec/bin/secStart firewall) evaluates to the following: "/opt/sec/bin/sec.pl -conf=/opt/sec/rules/firewall.rule -pid=/opt/sec/var/run/sec-firewall.pid -dump=/opt/sec/dump/sec-firewall.dump -debug=5 -syslog=local1 -intcontexts -intevents -input=-" If you need more information please let me know. best regards, Tom Zitat von Gergely Nagy <algernon@balabit.hu>:
Thomas Wollner <tw@wollner-net.de> writes:
I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected.
Ooh, this is interesting. Thanks for figuring this out, I'll see if I can figure out how to make it work with dash too. (Or if it's a bug in dash that should be fixed)
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
This is related to the bug I was describing earlier, so thanks all for pinning this down. One note: you should try to make things work for dash instead of bash for the long-term, because dash is 100% POSIX compliant, unlike bash, so things that work in dash should work in all POSIX shells; it is the least common denominator. On Wed, Nov 30, 2011 at 4:50 AM, Thomas Wollner <tw@wollner-net.de> wrote:
Hi,
currently I`m using syslog-ng 3.3.3 from madhouse package. In addition to the behaviour with the dash shell I found out that 3.3.3 does not handle signalling to scripts spawned by program destination.
Example:
in 3.1.3 this was a valid config statement:
destination d_raw_firewall_sec { program("`/opt/sec/bin/secStart firewall`" template(t_sec_template)); };
Have a look at the backticks inside the program.
After changing /bin/dash nach /bin/bash everything worked fine.
in 3.3.3. I have to change from backtick to $(...) style:
destination d_raw_firewall_sec { program("$(/opt/sec/bin/secStart firewall)" template(t_sec_template)); };
This works, but upon restart / reload of syslog-ng the "old" spawned scripts are not receiving the kill / term / hup signals and stay in the process list of the system until killed manually.
The $(/opt/sec/bin/secStart firewall) evaluates to the following:
"/opt/sec/bin/sec.pl -conf=/opt/sec/rules/firewall.rule -pid=/opt/sec/var/run/sec-firewall.pid -dump=/opt/sec/dump/sec-firewall.dump -debug=5 -syslog=local1 -intcontexts -intevents -input=-"
If you need more information please let me know.
best regards,
Tom
Zitat von Gergely Nagy <algernon@balabit.hu>:
Thomas Wollner <tw@wollner-net.de> writes:
I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected.
Ooh, this is interesting. Thanks for figuring this out, I'll see if I can figure out how to make it work with dash too. (Or if it's a bug in dash that should be fixed)
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thomas Wollner <tw@wollner-net.de> writes:
in 3.1.3 this was a valid config statement:
destination d_raw_firewall_sec { program("`/opt/sec/bin/secStart firewall`" template(t_sec_template)); };
Have a look at the backticks inside the program.
After changing /bin/dash nach /bin/bash everything worked fine.
in 3.3.3. I have to change from backtick to $(...) style:
destination d_raw_firewall_sec { program("$(/opt/sec/bin/secStart firewall)" template(t_sec_template)); };
This works, but upon restart / reload of syslog-ng the "old" spawned scripts are not receiving the kill / term / hup signals and stay in the process list of the system until killed manually.
The backtick thing is intended, I'm afraid: in syslog-ng 3.3, ` is used by the configuration parser, therefore, you can't use it as you did before. I don't think such usage was anticipated, otherwise this change would've been mentioned in the NEWS file. The solution is to use $(...) instead - that's supported by all POSIX compliant shells I know of, and even some others. As for the zombie processes, that's an unrelated thing, and a bug indeed, one which I'm working on. -- |8]
On Wed, 2011-11-30 at 11:50 +0100, Thomas Wollner wrote:
Hi,
currently I`m using syslog-ng 3.3.3 from madhouse package. In addition to the behaviour with the dash shell I found out that 3.3.3 does not handle signalling to scripts spawned by program destination.
Example:
in 3.1.3 this was a valid config statement:
destination d_raw_firewall_sec { program("`/opt/sec/bin/secStart firewall`" template(t_sec_template)); };
Have a look at the backticks inside the program.
After changing /bin/dash nach /bin/bash everything worked fine.
Backticks are now defined within syslog-ng too to substitute @defined variables and environment vars (and block arguments): @define logdir /var/log/remote destination d_file { file("`logdir`/$HOST/messages.log"); };
in 3.3.3. I have to change from backtick to $(...) style:
destination d_raw_firewall_sec { program("$(/opt/sec/bin/secStart firewall)" template(t_sec_template)); };
This works, but upon restart / reload of syslog-ng the "old" spawned scripts are not receiving the kill / term / hup signals and stay in the process list of the system until killed manually.
The $(/opt/sec/bin/secStart firewall) evaluates to the following:
"/opt/sec/bin/sec.pl -conf=/opt/sec/rules/firewall.rule -pid=/opt/sec/var/run/sec-firewall.pid -dump=/opt/sec/dump/sec-firewall.dump -debug=5 -syslog=local1 -intcontexts -intevents -input=-"
I know about an issue about program() destination and SIGHUP processing. For some reason it restarts a program one extra time during SIGHUP processing, which I haven't looked into yet. -- Bazsi
Thomas Wollner <tw@wollner-net.de> writes:
Hi,
I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected.
I believe I know why this happens, but I still need to test my suspicion: bash has an optimisation, that when a program is executed as "/bin/bash -c command", bash will exec it (as in, it will do the same as if one called "exec command" from the shell). Dash doesn't do this, and this might lead to the different behaviour. This will still need to be solved on the syslog-ng side, but I do not yet know how. I'll post a patch or more information as I progress further. -- |8]
On Fri, 2011-12-02 at 14:57 +0100, Gergely Nagy wrote:
Thomas Wollner <tw@wollner-net.de> writes:
Hi,
I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected.
I believe I know why this happens, but I still need to test my suspicion: bash has an optimisation, that when a program is executed as "/bin/bash -c command", bash will exec it (as in, it will do the same as if one called "exec command" from the shell).
Dash doesn't do this, and this might lead to the different behaviour.
This will still need to be solved on the syslog-ng side, but I do not yet know how. I'll post a patch or more information as I progress further.
Perhaps adding "exec" in front of the command line? syslog-ng needs a pid to the child in order to kill it. dash seems to fork an extra time, and doesn't propagate signals it receives to its children. syslog-ng is documented to spawn programs using the shell, however then it also needs signals. Another alternative is to send signals to the whole group, but that might cause problems if the executed program itself wants to handle SIGTERM gracefully by doing something with its children (w/o SIGTERM) -- Bazsi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, this issue is fixed in current 3.3.3 too. Scripts spawned by program destination are receiving the TERM signal upon restart. Thank you, Best regards, Tom On 18.12.2011 10:05, Balazs Scheidler wrote:
On Fri, 2011-12-02 at 14:57 +0100, Gergely Nagy wrote:
Thomas Wollner <tw@wollner-net.de> writes:
Hi,
I had a similar situation on debian squeeze a while ago. in debian 6 the default system shell is dash and syslog-ng hands over program destinations to the standard system shell. I changed the /bin/sh link from /bin/dash to /bin/bash and everything works as expected.
I believe I know why this happens, but I still need to test my suspicion: bash has an optimisation, that when a program is executed as "/bin/bash -c command", bash will exec it (as in, it will do the same as if one called "exec command" from the shell).
Dash doesn't do this, and this might lead to the different behaviour.
This will still need to be solved on the syslog-ng side, but I do not yet know how. I'll post a patch or more information as I progress further.
Perhaps adding "exec" in front of the command line? syslog-ng needs a pid to the child in order to kill it.
dash seems to fork an extra time, and doesn't propagate signals it receives to its children.
syslog-ng is documented to spawn programs using the shell, however then it also needs signals.
Another alternative is to send signals to the whole group, but that might cause problems if the executed program itself wants to handle SIGTERM gracefully by doing something with its children (w/o SIGTERM)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFPC2xPTCCRT+dccOYRAmn+AJ9XdvEt0BZAXOMttk4ROJ8VlUlsxQCeM5w6 IG+rnkwiRW3uFMKbEg5ZxfU= =CcTX -----END PGP SIGNATURE-----
participants (5)
-
Balazs Scheidler
-
Gergely Nagy
-
Martin Holste
-
Matt Zagrabelny
-
Thomas Wollner