chris packham <chris.packham@alliedtelesis.co.nz> wrote:

Well if it helps my Linux system uses /proc/kmsg (instead of /dev/klog). Have you tried using unix-stream("/dev/klog") instead of "file"?

Thanks, but, well, /dev/klog isn't a socket, it's a character special device, so "file" would be the only proper source to use. Using unix-stream hence fails. -Jan

On Wed, 2009-08-26 at 13:30 +0530, Jain, Vaibhav (GE Healthcare) wrote:

Hi,

I am using syslog-ng open source 3.0.3-1 ( rhel5.amd64) version. I am sending my log messages to a python parser(in program destination) to do some action on log messages.I want to know if my program application is down then can I know it in syslog-ng configuration? Because if my parser application is down then in that case I want to stop sending message to parser code because it's down, otherwise I will lose my log messages. And once the application comes up then I want to send the log messages.

syslog-ng starts the specified program and feeds it with log messages. When you script/program dies/exists syslog-ng detects it and tries to restart it. This way the program is always available to receive messages. If your program feeds the messages to any further application than it must take care of buffering the data, or stop reading from it's stdin, so syslog-ng can not feed it with logs. If you use flow-control in log statements than syslog-ng would also stop reading from it's log sources. M

# destinations destination d_parser { #Send message to parser program("python /opt/syslog-ng/etc/Parser.py"); };

log { source(s_network); destination(d_parser); };

Regards, Vaibhav ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

On Wed, 2009-08-26 at 15:43 +0530, Jain, Vaibhav (GE Healthcare) wrote:

Hi Marton,

Thanks for your quick response.

If program stops reading the message from stdin for some time, then in that time, will program lose the log message from syslog-ng ? Or syslog will starts feeding the logs from next to the last log mesg sent to program.

there's a buffer between your program and syslog-ng: the pipe buffer. This buffer is usually 8k in size, but that can depend from system to system. The flow of data between syslog-ng and your program works as follows: * syslog-ng writes messages to this buffer if there's space and suspends writing if the buffer is full * your program reads messages from the buffer if there's data This boils down to the following: * if your program is slower than syslog-ng, then this 8k of buffer becomes full quite fast, which syslog-ng notices and waits for the program to process more messages * if your program crashes, then this 8k of data can be lost, since syslog-ng has no way to differentiate between data sent to the buffer and data already processed by your application. Also, if there's a stall in the output processing in syslog-ng, it will suspend reading on its input, if you enable flow control on your logpath.

If my program is slower than incoming freq of syslog-ng logs then in that case will program lose the log mesgs from syslog-ng?

If syslog-ng stops reading logs from its source for some time then in that case will syslog-ng lose the mesg from source ? If source is tcp port.

Note: I am using open source version of syslog-ng (3.0.3-1 On rhel5)

-- Bazsi

Hi, Thanks for your quick response. As per my understaning the buffering facility is only supported in the premium version but I am using the Open source verison that's why I asked these questions. Does syslog-ng open source supports the buffering facility for log messages ? If syslog-ng server is down then does syslog-ng client buffers the log message? What it the maximum limit of buffer size and how to set it in syslog-ng open source configuration file? Is it log_fifo_size() ? I am using the following code in logrotate.d file to send the HUP signal to syslog-ng. Let me know this is correct way to send the HUP signal to syslog-ng or not ? Do I need to add any configuration in syslog-ng ? Is there any chance to loss the data? /var/log/*.log { postrotate /bin/kill -HUP `cat /var/run/syslogd-ng.pid 2> /dev/null` 2> /dev/null || true endscript } Regards, VJ -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Sandor Geller Sent: Wednesday, September 02, 2009 7:24 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Problems with syslog-ng Hi, On Wed, Sep 2, 2009 at 5:46 AM, Jain, Vaibhav (GE Healthcare)<Vaibhav.Jain@ge.com> wrote:

Hello,

Can anybody help me ? This is the first time I am not getting any response from this forum. Please help!

Don't get this wrong but your questions were too generic. syslog-ng uses internal buffering for logs so when there is a problem with a destination then syslog-ng keeps messages in the buffer. When the buffer gets overrun then syslog-ng starts dropping old messages. syslog-ng OSE doesn't deal with log rotation on its own. It just opens a "file" and then uses the given fd. It doesn't try to close and reopen destinations as long as writes succeed. So If you rename the file and/or create another with the same name then syslog-ng won't notice it and will happily write to the same fd, just like unix apps usually do. It's a common practice to send a HUP signal to the syslog daemon when log rotation occured. hth, Sandor ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Sandor Geller wrote:

Hi,

On Thu, Sep 3, 2009 at 6:30 AM, Jain, Vaibhav (GE Healthcare)<Vaibhav.Jain@ge.com> wrote:

...

Hi,

Thanks for your quick response.

As per my understaning the buffering facility is only supported in the premium version but I am using the Open source verison that's why I asked these questions.

syslog-ng OSE has in-memory buffering only. The contents of the buffers will get lost when syslog-ng stops or reinitialises itself (when receiving a HUP signal for example). In contrast syslog-ng PE has persistent on-disk buffering in addition to avoid losing logs.

Can you confirm that a HUP will loose the buffers please? I was under the impression that a HUP *only* caused syslog-ng to close and reopen its source and destinations.

...

Does syslog-ng open source supports the buffering facility for log messages ? If syslog-ng server is down then does syslog-ng client buffers the log message? What it the maximum limit of buffer size and how to set it in syslog-ng open source configuration file? Is it log_fifo_size() ?

Yes, it's log_fifo_size. I think only the available physical memory is the real limitation here...

...

I am using the following code in logrotate.d file to send the HUP signal to syslog-ng. Let me know this is correct way to send the HUP signal to syslog-ng or not ? Do I need to add any configuration in syslog-ng ? Is there any chance to loss the data?

/var/log/*.log { postrotate /bin/kill -HUP `cat /var/run/syslogd-ng.pid 2> /dev/null` 2> /dev/null || true endscript }

I'd limit the number of signals sent to syslog-ng to 1. Distros usually use a single logrotate file where there are multiple logs rotated, and send the HUP signal only when the last log has been rotated. I usually add delaycompress to all rotate configs as well to avoid silent data loss (which could happen when logrotate compresses the logfile but syslog-ng still sends messages to the file which is being compressed, there is a race between the two apps).

Regards,

Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

-- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Unix Services, University Systems, University of Victoria