October 2006 - syslog-ng - lists.balabit.hu

Idea for streaming logs to my workstation as well as logserver
by Hari Sekhon 31 Oct '06

31 Oct '06

I have a cunning idea. When I first set up a logserver, I also had it log to tty12, and a monitor next to my desk of the server with the network logs streaming down the screen. This was very interesting and I managed to pick up quite a lot about my network by noticing when a sudden screen full of information would whizz by instead of the usual incremental logs coming in. I would like to implement something similar by having the logserver also send on it's logs to my linux workstation by way of udp. I could then have my local syslog-ng listen for this udp and write it to a pipe (similar to the way you integrate mysql with syslog-ng). I would then like to have root-tail read from this pipe and output to my X background so that I have a continuous stream of network logs in the background. Are there any pitfalls that you can think of when doing this? I don't think this will risk filling up the bit bucket on the logserver since the logs are immediately sent on via udp, "spray and pray". If they don't get to my workstation for any reason, no harm done, it won't clog up the logserver. I guess the root-tail will take a bit out of my workstation having to draw on the X background all the time but it's a fast machine. I believe that it would block my local syslog-ng on my workstation if the root-tail were to stop reading from it. Which brings me to my next question: Is it better to do root-tail /var/log/logstream.pipe or root-tail - < /var/log/logstream.pipe Since I'm not sure that the first will take the actual throughput away from the pipe to stop the logger from blocking on the sending side. Also, this would require that the my X session not be closed otherwise the root-tail wouldn't be able to run to take away the logs from the other side of the pipe and the local logger would block again. All feedback welcome. Thanks. -h -- Hari Sekhon

2 3

Re: [syslog-ng] runaway programs started by syslog-ng
by Dev Man 30 Oct '06

30 Oct '06

A followup... On a test server, I have noticed that when the system reboots there is one instance of syslog-ng but two instances of the program being executed. There should only be one. If I do a /etc/init.d/syslog-ng restart, then I end up with just one instance of the program as expected, and the system appears to be fine. Any ideas?

3 2

2.0RC4 make warning on OS X 10.4.8 PPC G4
by Mr. Brian Opitz 30 Oct '06

30 Oct '06

System is OS X 10.4.8, Xcode 2.4, PPC G4(2.7), configure --enable-debug --enable-dynamic-linking I am seeing the following warnings during compiling 2.0RC4: logmsg.c: In function 'log_msg_parse': logmsg.c:149: warning: pointer targets in assignment differ in signedness logmsg.c:212: warning: pointer targets in passing argument 2 of 'g_string_assign_len' differ in signedness logmsg.c:214: warning: pointer targets in assignment differ in signedness logmsg.c:232: warning: pointer targets in passing argument 1 of 'strlen' differ in signedness logmsg.c:271: warning: pointer targets in passing argument 2 of 'g_string_assign_len' differ in signedness logmsg.c:309: warning: pointer targets in assignment differ in signedness logmsg.c:313: warning: pointer targets in assignment differ in signedness logmsg.c:344: warning: pointer targets in assignment differ in signedness logmsg.c:378: warning: pointer targets in assignment differ in signedness logmsg.c:389: warning: pointer targets in assignment differ in signedness logmsg.c:401: warning: pointer targets in assignment differ in signedness logmsg.c:413: warning: pointer targets in assignment differ in signedness logmsg.c:435: warning: pointer targets in assignment differ in signedness logmsg.c:440: warning: pointer targets in assignment differ in signedness logmsg.c:445: warning: pointer targets in passing argument 2 of 'g_string_assign_len' differ in signedness and: affile.c: In function 'affile_dd_init': affile.c:456: warning: comparison is always false due to limited range of data type affile.c:462: warning: comparison is always false due to limited range of data type and: afuser.c: In function 'afuser_dd_queue': afuser.c:57: warning: implicit declaration of function 'getutent' afuser.c:57: warning: assignment makes pointer from integer without a cast and finally: gsockaddr.c: In function 'g_accept': gsockaddr.c:116: warning: pointer targets in passing argument 3 of 'accept' differ in signedness Can these safely be ignored or is there something that needs to be changed to make them go away? --

5 5

Re: [syslog-ng] runaway programs started by syslog-ng
by Dev Man 27 Oct '06

27 Oct '06

No, the program does not fork or execute any other programs. It just processes input from syslog and does some inserts into a database. Due to the out of control restarts, they end up hogging up database connections, too. :( --===============0443184984== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C6FA0C.AE82AFEE" This is a multi-part message in MIME format. Simple question which may help: does your program fork() or daemon() into a "background" process? This would explain the behavior you're seeing, and should be relatively easy to work around. Hope that helps. Good luck, Peter Nahas MRV Communications - Boston Division ________________________________ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Dev Man Sent: Friday, October 27, 2006 3:38 PM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] runaway programs started by syslog-ng Hi Bazsi, How would having that switch help? The problem is we do want it to automatically restart. It just appears that there are some scenarios which we haven't figure out.. in which the program gets restarted repeatedly leaving apparently multiple instances running. How do you determine when to restart? Thanks! On Fri, 2006-10-27 at 10:59 -0700, Dev Man wrote: > > Hi, > > We have experienced cases of syslog-ng apparently launching multiple > instances of the same program unexpectedly, causing the machine to > become unusable. Some of the launched processes can end up defunct, > but not all appeared to be when we checked the running processes. > > Isn't syslog-ng only supposed to start a program if it was terminated? > > In one case, we were seeing a constant stream of "duplicate stats > counter" messages like: > syslog.1:Oct 26 06:27:10 in-mta syslog-ng[29833]: Duplicate stats > counter; counter='program(/home/readlog/readlog.php)' > > One machine was using syslog-ng version 1.9.11 and another version > 2.0rc1. 2.0.x (and 1.9.x which was development versions for 2.0) restart program() destinations by default. I'll add a switch to disable that behaviour as it is hardwired at the moment. -- Bazsi [Attachment #3 (text/html)] <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=Content-Type content="text/html; charset=us-ascii"> <META content="MSHTML 6.00.2900.2963" name=GENERATOR></HEAD> <BODY> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2>Simple question which may help: does your program fork() or daemon() into a "background" process?  This would explain the behavior you're seeing, and should be relatively easy to work around.</FONT></SPAN></DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2></FONT></SPAN> </DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2>Hope that helps.</FONT></SPAN></DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2></FONT></SPAN> </DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2></FONT></SPAN> </DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2>Good luck,</FONT></SPAN></DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2></FONT></SPAN> </DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2>Peter Nahas</FONT></SPAN></DIV> <DIV dir=ltr align=left><SPAN class=510391021-27102006><FONT face=Arial color=#0000ff size=2>MRV Communications - Boston Division</FONT></SPAN><BR></DIV> <DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left> <HR tabIndex=-1> <FONT face=Tahoma size=2><B>From:</B> syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] <B>On Behalf Of </B>Dev Man<BR><B>Sent:</B> Friday, October 27, 2006 3:38 PM<BR><B>To:</B> syslog-ng(a)lists.balabit.hu<BR><B>Subject:</B> Re: [syslog-ng] runaway programs started by syslog-ng<BR></FONT><BR></DIV> <DIV></DIV> <DIV> </DIV> <DIV>Hi Bazsi,</DIV> <DIV> </DIV> <DIV>How would having that switch help?</DIV> <DIV>The problem is we do want it to automatically restart.  It just appears that there are some scenarios which we haven't figure out.. in which the program gets restarted repeatedly leaving apparently multiple instances running. </DIV> <DIV> </DIV> <DIV>How do you determine when to restart?</DIV> <DIV> </DIV> <DIV>Thanks!</DIV> <DIV><PRE>On Fri, 2006-10-27 at 10:59 -0700, Dev Man wrote: > > Hi, > > We have experienced cases of syslog-ng apparently launching multiple > instances of the same program unexpectedly, causing the machine to > become unusable. Some of the launched processes can end up defunct, > but not all appeared to be when we checked the running processes. > > Isn't syslog-ng only supposed to start a program if it was terminated? > > In one case, we were seeing a constant stream of "duplicate stats > counter" messages like: > syslog.1:Oct 26 06:27:10 in-mta syslog-ng[29833]: Duplicate stats > counter; counter='program(/home/readlog/readlog.php)' > > One machine was using syslog-ng version 1.9.11 and another version > 2.0rc1. 2.0.x (and 1.9.x which was development versions for 2.0) restart program() destinations by default. I'll add a switch to disable that behaviour as it is hardwired at the moment. -- Bazsi</PRE></DIV></BODY></HTML> _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

RE: [syslog-ng] runaway programs started by syslog-ng
by Peter Nahas 27 Oct '06

27 Oct '06

Simple question which may help: does your program fork() or daemon() into a "background" process? This would explain the behavior you're seeing, and should be relatively easy to work around. Hope that helps. Good luck, Peter Nahas MRV Communications - Boston Division ________________________________ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Dev Man Sent: Friday, October 27, 2006 3:38 PM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] runaway programs started by syslog-ng Hi Bazsi, How would having that switch help? The problem is we do want it to automatically restart. It just appears that there are some scenarios which we haven't figure out.. in which the program gets restarted repeatedly leaving apparently multiple instances running. How do you determine when to restart? Thanks! On Fri, 2006-10-27 at 10:59 -0700, Dev Man wrote: > > Hi, > > We have experienced cases of syslog-ng apparently launching multiple > instances of the same program unexpectedly, causing the machine to > become unusable. Some of the launched processes can end up defunct, > but not all appeared to be when we checked the running processes. > > Isn't syslog-ng only supposed to start a program if it was terminated? > > In one case, we were seeing a constant stream of "duplicate stats > counter" messages like: > syslog.1:Oct 26 06:27:10 in-mta syslog-ng[29833]: Duplicate stats > counter; counter='program(/home/readlog/readlog.php)' > > One machine was using syslog-ng version 1.9.11 and another version > 2.0rc1. 2.0.x (and 1.9.x which was development versions for 2.0) restart program() destinations by default. I'll add a switch to disable that behaviour as it is hardwired at the moment. -- Bazsi

1 0

Re: [syslog-ng] runaway programs started by syslog-ng
by Dev Man 27 Oct '06

27 Oct '06

Hi Bazsi, How would having that switch help? The problem is we do want it to automatically restart. It just appears that there are some scenarios which we haven't figure out.. in which the program gets restarted repeatedly leaving apparently multiple instances running. How do you determine when to restart? Thanks! On Fri, 2006-10-27 at 10:59 -0700, Dev Man wrote: > > Hi, > > We have experienced cases of syslog-ng apparently launching multiple > instances of the same program unexpectedly, causing the machine to > become unusable. Some of the launched processes can end up defunct, > but not all appeared to be when we checked the running processes. > > Isn't syslog-ng only supposed to start a program if it was terminated? > > In one case, we were seeing a constant stream of "duplicate stats > counter" messages like: > syslog.1:Oct 26 06:27:10 in-mta syslog-ng[29833]: Duplicate stats > counter; counter='program(/home/readlog/readlog.php)' > > One machine was using syslog-ng version 1.9.11 and another version > 2.0rc1. 2.0.x (and 1.9.x which was development versions for 2.0) restart program() destinations by default. I'll add a switch to disable that behaviour as it is hardwired at the moment. -- Bazsi

1 0

runaway programs started by syslog-ng
by Dev Man 27 Oct '06

27 Oct '06

Hi, We have experienced cases of syslog-ng apparently launching multiple instances of the same program unexpectedly, causing the machine to become unusable. Some of the launched processes can end up defunct, but not all appeared to be when we checked the running processes. Isn't syslog-ng only supposed to start a program if it was terminated? In one case, we were seeing a constant stream of "duplicate stats counter" messages like: syslog.1:Oct 26 06:27:10 in-mta syslog-ng[29833]: Duplicate stats counter; counter='program(/home/readlog/readlog.php)' One machine was using syslog-ng version 1.9.11 and another version 2.0rc1. Any suggestions? Thanks. D

2 1

Re: [syslog-ng] use syslog-ng to monitor just couple of processes
by Hari Sekhon 27 Oct '06

27 Oct '06

I've not used rsyslog myself, but it seems a lot closer to the old syslog than syslog-ng which is quite different and very flexible. Rsyslog seems to do most of the things that syslog-ng does so I guess it might be down to which you personally prefer. Syslog-ng seems more prevalent, and although rsyslog seems to have built in mysql support, whereas syslog-ng uses a tiny glue shell script, the syslog-ng way works well enough too. I run my central logserver this way and it is reliable. Neither of these things really support web interfaces to my knowledge. My web interface is an interface to mysql, nothing to do with syslog-ng.... -h Hari Sekhon Tiger Peng wrote: > Hi, Hari: > Thanks a lot, since I also consider to dump the > syslog-ng log to database (MySQL or Oracle), do you > think it is better to choose the syslog-ng, or > rsyslog, it looks like the syslog-ng has better web > interface supporting, but rsyslog has the better built > in MySQL database support. > Definitely, I will take your advice to read through > all the syslog-ng documents and FAQ. Thanks again. > > > > David > > = = = Original message = = = > > ? > > You don't use syslog-ng in this way. For what you want > to do you should make your application output the > information you want to the standard syslog and use > syslog-ng as your logger. Then use filters to extract > the messages you want the launch an external program > to email them off to you or something. That is what I > did. A few lines of code, some regex filtering with > syslog-ng definitions is all it takes. > > Now you have all the pointers, you can read the docs > on filtering, program(), etc. > Although it sounds like you don't have a good grasp of > the whole logging thing yet so my advice is to read > all the syslog-ng docs and faq. They are worth the > read. > > -h > > > > Hari Sekhon > > Tiger Peng wrote: > > I have two questions about how to use the > syslog-ng to monitor just couple of processes. > In my case, I try to use some kind of utility to > just monitor 4 to 8 different processes, some of them > are system resource, but some of them are new > developed processes. Here are my questions. > > 1. can I start a seperate syslog-ng, I mean if I can > keep syslogd running, but start another syslog-ng > process to dedicate the service for my special > purpose. > > 2. I guess syslog-ng has some kind of filter which can > filter the information, but how can I expand it > ability to receive some specific information or more > field, such as message ID. Is this configurable, or I > must modify the source to support it. > > Thanks. > > > > David > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at > http://www.campin.net/syslog-ng/faq.html > > ___________________________________________________________ > Sent by ePrompter, the premier email notification > software. > Free download at http://www.ePrompter.com. > >

1 0

use syslog-ng to monitor just couple of processes
by Tiger Peng 27 Oct '06

27 Oct '06

I have two questions about how to use the syslog-ng to monitor just couple of processes. In my case, I try to use some kind of utility to just monitor 4 to 8 different processes, some of them are system resource, but some of them are new developed processes. Here are my questions. 1. can I start a seperate syslog-ng, I mean if I can keep syslogd running, but start another syslog-ng process to dedicate the service for my special purpose. 2. I guess syslog-ng has some kind of filter which can filter the information, but how can I expand it ability to receive some specific information or more field, such as message ID. Is this configurable, or I must modify the source to support it. Thanks. David

2 1

beginner's question about how to use Java to write log to syslog-ng
by Tiger Peng 26 Oct '06

26 Oct '06

I am the beginner of the syslog-ng, and have a very basic question. How to use Java to write the system log to syslog-ng? Is the same syslogger wrapper can be used here? Another suggestion is that the list didn't have the search ability, it should be very helpful to have it, I believe a lot of people do like to see it. Thanks. David

2 1