Re: [syslog-ng] destination program is killed due to broken pipe
well the script on the surface sounds like a good simple idea, the simple way is not always the best or most relyable way. There are several better options open to you that should work better and shouldn't be too hard to impliment. The first is to write to a log file then use something like a "tail -f" in your script. Another option open to you is if you know how to write unix network program's in any languages (I use perl for this kind of thing my self) it wouldn't be difficult to write a simple application read from a socket, or even udp syslog on an alternitive port on the loopback address. Syslog is a relitivly simple protocal to write a listener for and there are several perl modules that can do most of the work for you. ----------------------- Sent from my Treo(r) smartphone -----Original Message----- From: "midnight codder" <midnight.codder@gmail.com> Date: Tuesday, Aug 19, 2008 5:08 pm Subject: [syslog-ng] destination program is killed due to broken pipe To: syslog-ng@lists.balabit.huReply-To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> I am trying to redirect all log messages to a destination program (written in java). This did not do much so I redirected all messages to a single catch-all log file AND also to a bash shell script that runs an endless loop, reads a line and spit it out to a text file. The shell script is killed by syslog-ng after the 5th message. The catch-all file, however, keeps getting all of the messages (that's how I know who is killing who). Here is the config file for syslog-ng: options { ts_format(iso); frac_digits(3); }; template bds_template{ template('<pri=$PRI> <date=$ISODATE> <host=$HOST> <msg=$MSG>\n'); template_escape(no); }; source src{ internal(); unix_stream('/dev/log'); unix-dgram('/dev/log'); file('/proc/kmsg' log_prefix('kernel:')); udp(); tcp(); }; destination test_adapter{ program('bash /usr/local/bds/logger/syslogAdapterTest.sh' template(bds_template)); }; log { source(src); destination(test_adapter); }; destination catchAll{ file('/var/log/syslog-ng-catch-all.log', template(bds_template)); }; log { source(src); destination(catchAll); }; Here is the bash script: #!/bin/bash file='/root/syslogAdapterTest' counter=1 limit=3 while true do read line echo '$line' > $file.$counter ((counter++)) done and finally the content of the catch-all file, in blue are the log messages that the shell script still captures and spits to text files, in red are the messages are the messages the shell does not get: <priE> <date 08-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Server license found, starting in server mode; customer='University of Illinois', serial='49544f30-573e-11dd-9892-0007e9195011', limit='25'> <priF> <date 08-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Starting destination program; cmdline='bash /usr/local/bds/logger/syslogAdapterTest.sh'> <priE> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: syslog-ng starting up; version='2.1.11a'> <priF> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Initializing destination file writer; template='/var/log/syslog-ng-catch-all.log', filename='/var/log/syslog-ng-catch-all.log'> <priE> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: License validity expires in one week, logging service will terminate at that point; error='License outside the validity period (option=syslog-ng, not-before 08/07/21, not-after 08/08/21)'> <priC> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <priF> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Sending child a TERM signal; child_pid='27972'> Any idea?
Thanks for all of those alternatives. Allow me to rephrase my question (any idea): Does anyone have any idea as to why I am getting the broken pipe? The script I am using now is for the sake of simplifying things so I can pin point where troubles begin. We are not going to use such a script in production. The original java program that was supposed to be the destination is much more complicated, although in a sense it still reads the log messages off of std in, just like the script. (it is the handling of the logs afterwards that is complex). I shall investigate into using one of the alternatives, however - I will still be happy to know what might have went wrong. Thanks. On Wed, Aug 20, 2008 at 12:55 AM, <prmarino1@gmail.com> wrote:
well the script on the surface sounds like a good simple idea, the simple way is not always the best or most relyable way. There are several better options open to you that should work better and shouldn't be too hard to impliment. The first is to write to a log file then use something like a "tail -f" in your script. Another option open to you is if you know how to write unix network program's in any languages (I use perl for this kind of thing my self) it wouldn't be difficult to write a simple application read from a socket, or even udp syslog on an alternitive port on the loopback address. Syslog is a relitivly simple protocal to write a listener for and there are several perl modules that can do most of the work for you. ----------------------- Sent from my Treo(r) smartphone
-----Original Message----- From: "midnight codder" <midnight.codder@gmail.com> Date: Tuesday, Aug 19, 2008 5:08 pm Subject: [syslog-ng] destination program is killed due to broken pipe To: syslog-ng@lists.balabit.huReply-To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
I am trying to redirect all log messages to a destination program (written in java). This did not do much so I redirected all messages to a single catch-all log file AND also to a bash shell script that runs an endless loop, reads a line and spit it out to a text file. The shell script is killed by syslog-ng after the 5th message. The catch-all file, however, keeps getting all of the messages (that's how I know who is killing who).
Here is the config file for syslog-ng:
options { ts_format(iso); frac_digits(3); };
template bds_template{ template('<pri=$PRI> <date=$ISODATE> <host=$HOST> <msg=$MSG>\n'); template_escape(no); };
source src{ internal(); unix_stream('/dev/log'); unix-dgram('/dev/log'); file('/proc/kmsg' log_prefix('kernel:')); udp(); tcp(); };
destination test_adapter{ program('bash /usr/local/bds/logger/syslogAdapterTest.sh' template(bds_template)); };
log { source(src); destination(test_adapter); };
destination catchAll{ file('/var/log/syslog-ng-catch-all.log', template(bds_template)); };
log { source(src); destination(catchAll); };
Here is the bash script:
#!/bin/bash
file='/root/syslogAdapterTest' counter=1 limit=3
while true do read line echo '$line' > $file.$counter ((counter++)) done
and finally the content of the catch-all file, in blue are the log messages that the shell script still captures and spits to text files, in red are the messages are the messages the shell does not get:
<priE> <date 08-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Server license found, starting in server mode; customer='University of Illinois', serial='49544f30-573e-11dd-9892-0007e9195011', limit='25'> <priF> <date 08-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Starting destination program; cmdline='bash /usr/local/bds/logger/syslogAdapterTest.sh'> <priE> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: syslog-ng starting up; version='2.1.11a'> <priF> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Initializing destination file writer; template='/var/log/syslog-ng-catch-all.log', filename='/var/log/syslog-ng-catch-all.log'> <priE> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: License validity expires in one week, logging service will terminate at that point; error='License outside the validity period (option=syslog-ng, not-before 08/07/21, not-after 08/08/21)'> <priC> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <priF> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Sending child a TERM signal; child_pid='27972'>
Any idea?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2008-08-20 at 09:10 -0500, midnight codder wrote:
Thanks for all of those alternatives. Allow me to rephrase my question (any idea): Does anyone have any idea as to why I am getting the broken pipe? The script I am using now is for the sake of simplifying things so I can pin point where troubles begin. We are not going to use such a script in production. The original java program that was supposed to be the destination is much more complicated, although in a sense it still reads the log messages off of std in, just like the script. (it is the handling of the logs afterwards that is complex). I shall investigate into using one of the alternatives, however - I will still be happy to know what might have went wrong.
<priC> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <priF> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Sending child a TERM signal; child_pid='27972'>
Your script terminates for some reason, it's not syslog-ng that kills your process, it is your script that dies. please run syslog-ng under strace -f (to follow forks) and see why your shell script exits. -- Bazsi
On Thu, Aug 21, 2008 at 7:25 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2008-08-20 at 09:10 -0500, midnight codder wrote:
Thanks for all of those alternatives. Allow me to rephrase my question (any idea): Does anyone have any idea as to why I am getting the broken pipe? The script I am using now is for the sake of simplifying things so I can pin point where troubles begin. We are not going to use such a script in production. The original java program that was supposed to be the destination is much more complicated, although in a sense it still reads the log messages off of std in, just like the script. (it is the handling of the logs afterwards that is complex). I shall investigate into using one of the alternatives, however - I will still be happy to know what might have went wrong.
<priC> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <priF> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Sending child a TERM signal; child_pid='27972'>
Your script terminates for some reason, it's not syslog-ng that kills your process, it is your script that dies.
please run syslog-ng under strace -f (to follow forks) and see why your shell script exits.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Balazs, I can't seem to find the command line option that you've mentioned. there is -F (running int he foreground) and -f <config file name>. I don't see any command line option that refers to following forks. I am running syslog-ng with -v and -d to get as many as messages as possible. The good news, however, are that the script now is running successfully. (I had a typo, as usually, the mistake is so obvious it cannot be noticed) But my java program still fails to intercept any message. For any of you java driven syslog-ng users out there, please review this bit of code, perhaps you can point out where I went wrong. Here is the java code, it is a small main method: static public void main(String[] args) throws IOException { int counter = 0; Scanner scanner = new Scanner(System.in); while (scanner.hasNextLine()) { counter++; String syslog = scanner.nextLine(); FileWriter writer = new FileWriter( new File( System.getProperty("user.home") + "/" + SyslogProgramDestination.class.getSimpleName() + "_" + counter + ".log")); try { writer.write(syslog + "\n"); } catch (Exception e) { e.printStackTrace(); } finally { writer.close(); } } } }
I realized strace is a unix command and not an option of syslog-ng. Sorry about that. After running syslog-ng with strace I found out, what was already known - syslog-ng is trying to write to the destination program but getting a broken pipe error: This is from the output of strace: 598 write(13, "<pri=43> <date=2008-08-27T15:54:"..., 155) = -1 EPIPE (Broken pipe) 598 --- SIGPIPE (Broken pipe) @ 0 (0) --- 598 kill(600, SIGTERM) = 0 And this is from the catch-all destination file: <pri=46> <date=2008-08-27T15:54:35.092-05:00> <host=oracle-test> <msg=syslog-ng[596]: Starting destination program; cmdline='bash /usr/local/bds/logger/syslogAdapterProgram.sh'> <pri=45> <date=2008-08-27T15:54:35.101-05:00> <host=oracle-test> <msg=syslog-ng[598]: syslog-ng starting up; version='2.1.11a'> <pri=46> <date=2008-08-27T15:54:35.102-05:00> <host=oracle-test> <msg=syslog-ng[598]: Initializing destination file writer; template='/var/log/syslog-ng-catch-all.log', filename='/var/log/syslog-ng-catch-all.log'> <pri=43> <date=2008-08-27T15:54:35.174-05:00> <host=oracle-test> <msg=syslog-ng[598]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <pri=46> <date=2008-08-27T15:54:35.174-05:00> <host=oracle-test> <msg=syslog-ng[598]: Sending child a TERM signal; child_pid='597'> I guess my question is: are there any configuration options I was over looking, that may resolve this? Or am I not using the Scanner the right way? what can cause a broken IO pipe? Thanks. On Thu, Aug 21, 2008 at 7:25 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2008-08-20 at 09:10 -0500, midnight codder wrote:
Thanks for all of those alternatives. Allow me to rephrase my question (any idea): Does anyone have any idea as to why I am getting the broken pipe? The script I am using now is for the sake of simplifying things so I can pin point where troubles begin. We are not going to use such a script in production. The original java program that was supposed to be the destination is much more complicated, although in a sense it still reads the log messages off of std in, just like the script. (it is the handling of the logs afterwards that is complex). I shall investigate into using one of the alternatives, however - I will still be happy to know what might have went wrong.
<priC> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <priF> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Sending child a TERM signal; child_pid='27972'>
Your script terminates for some reason, it's not syslog-ng that kills your process, it is your script that dies.
please run syslog-ng under strace -f (to follow forks) and see why your shell script exits.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2008-08-27 at 16:07 -0500, midnight codder wrote:
I realized strace is a unix command and not an option of syslog-ng. Sorry about that.
After running syslog-ng with strace I found out, what was already known - syslog-ng is trying to write to the destination program but getting a broken pipe error:
This is from the output of strace:
598 write(13, "<pri=43> <date=2008-08-27T15:54:"..., 155) = -1 EPIPE (Broken pipe) 598 --- SIGPIPE (Broken pipe) @ 0 (0) --- 598 kill(600, SIGTERM) = 0
And this is from the catch-all destination file: <pri=46> <date=2008-08-27T15:54:35.092-05:00> <host=oracle-test> <msg=syslog-ng[596]: Starting destination program; cmdline='bash /usr/local/bds/logger/syslogAdapterProgram.sh'> <pri=45> <date=2008-08-27T15:54:35.101-05:00> <host=oracle-test> <msg=syslog-ng[598]: syslog-ng starting up; version='2.1.11a'> <pri=46> <date=2008-08-27T15:54:35.102-05:00> <host=oracle-test> <msg=syslog-ng[598]: Initializing destination file writer; template='/var/log/syslog-ng-catch-all.log', filename='/var/log/syslog-ng-catch-all.log'> <pri=43> <date=2008-08-27T15:54:35.174-05:00> <host=oracle-test> <msg=syslog-ng[598]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <pri=46> <date=2008-08-27T15:54:35.174-05:00> <host=oracle-test> <msg=syslog-ng[598]: Sending child a TERM signal; child_pid='597'>
I guess my question is: are there any configuration options I was over looking, that may resolve this? Or am I not using the Scanner the right way? what can cause a broken IO pipe?
Broken pipe is caused by either your application closing its stdin, or exiting prematurely. You can also use strace on your java program to see what happens. e.g. use strace in the program() destination like this: strace -o /tmp/strace.out -f -s 256 java -cp $CLASSPATH /path/to/java/file Then look at strace.out and see what it does with its file descriptor 0 (which is stdin) -- Bazsi
With a lot of help from unix flavored co-workers I got this issue solved! In short the problem was doubled: Fist the way I configured the template of the syslog messages included characters that the bash shell interperted them as a signal to close the input stream. Secondly the bash script was creating the classpath relatively to where the script resides. But syslog-ng runs the script from where ever syslog-ng is, which resulted in the classpath being in-valid or just plain worthless. So to solve this I have modified the script to use exec to call java, this way the input stream is not going through the bash shell, or at least that's how it was explained to me. After this was done the program was still terminating, but this time on its own, because the main class was not found int he class path. So I modified the script again and added a prefix to the classpath relative elementes - with a full path to where the script resides. And now it all works. Thanks for all the strace information it really helped a LOT to pinpoint where the problem is. On Thu, Aug 28, 2008 at 3:01 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2008-08-27 at 16:07 -0500, midnight codder wrote:
I realized strace is a unix command and not an option of syslog-ng. Sorry about that.
After running syslog-ng with strace I found out, what was already known - syslog-ng is trying to write to the destination program but getting a broken pipe error:
This is from the output of strace:
598 write(13, "<pri=43> <date=2008-08-27T15:54:"..., 155) = -1 EPIPE (Broken pipe) 598 --- SIGPIPE (Broken pipe) @ 0 (0) --- 598 kill(600, SIGTERM) = 0
And this is from the catch-all destination file: <pri=46> <date=2008-08-27T15:54:35.092-05:00> <host=oracle-test> <msg=syslog-ng[596]: Starting destination program; cmdline='bash /usr/local/bds/logger/syslogAdapterProgram.sh'> <pri=45> <date=2008-08-27T15:54:35.101-05:00> <host=oracle-test> <msg=syslog-ng[598]: syslog-ng starting up; version='2.1.11a'> <pri=46> <date=2008-08-27T15:54:35.102-05:00> <host=oracle-test> <msg=syslog-ng[598]: Initializing destination file writer; template='/var/log/syslog-ng-catch-all.log', filename='/var/log/syslog-ng-catch-all.log'> <pri=43> <date=2008-08-27T15:54:35.174-05:00> <host=oracle-test> <msg=syslog-ng[598]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <pri=46> <date=2008-08-27T15:54:35.174-05:00> <host=oracle-test> <msg=syslog-ng[598]: Sending child a TERM signal; child_pid='597'>
I guess my question is: are there any configuration options I was over looking, that may resolve this? Or am I not using the Scanner the right way? what can cause a broken IO pipe?
Broken pipe is caused by either your application closing its stdin, or exiting prematurely.
You can also use strace on your java program to see what happens.
e.g. use strace in the program() destination like this:
strace -o /tmp/strace.out -f -s 256 java -cp $CLASSPATH /path/to/java/file
Then look at strace.out and see what it does with its file descriptor 0 (which is stdin)
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (3)
-
Balazs Scheidler
-
midnight codder
-
prmarino1＠gmail.com