[syslog-ng]syslog-ng loses messages
Dave Johnson
syslog-ng@lists.balabit.hu
Thu, 7 Oct 2004 10:47:14 -0500
We have a couple solaris boxes that do logging with pipes and we log a
large amount of data, thats why I'm interested in these tests. I can
see if I can setup a quick testbed next week.
**********
Otherwise, I'm wondering if the packets are throttled, what happens.. IE:
---------------------------
cat > /tmp/smallsleep.c <<ENDHERE
#include <unistd.h>
main () {
usleep(50000); /* 20 hz */
}
ENDHERE
gcc -o /tmp/smallsleep /tmp/smallsleep.c
-----------------------------
in the do -> done segment of your for loop add `/tmp/smallsleep`
How does this impact lost messages?
*****************
On Mon, 4 Oct 2004 16:43:31 +0100, Aaron watkins
<aaron.j.watkins@gmail.com> wrote:
> Dave,
>
> > * What are the numbers of lines that get dropped in each of those tests?
>
> If I kill (-TERM or -9) the receiving server while messages are not
> being written to the input pipe, I will lose message 0 and message 1.
>
> When I kill -TERM while writing to the input pipe:
> Test 1: Messages 27->34 are lost (8 in total)
> Test 2: Messages 24->31 are lost (8 in total)
>
> When I kill -9 while writing to the input pipe:
> Test 1: Messages 35->38 are lost (4 in total)
> Test 2: Messages 23->29 are lost (7 in total)
>
>
> > * What happens in your test if you change machine2 fifo's to 0?
>
> I presume you mean the log_fifo_size option... When I set this to 0,
> the following messages got through:
> Iteration 1: Message 0,1,14,28
> Iteration 2: Message 0,1,12,21,32
> Iteration 3: Message 0,1,12,21,30,39
>
> That is to say, in every iteration, around 37 messages were lost. And
> this was with the system operating normally (ie. without killing the
> machine2's syslog-ng).
>
> Any ideas?
>
> Thanks,
> Aaron
>
>
>
>
> >
> > On Fri, 1 Oct 2004 12:38:56 +0100, Aaron watkins
> > <aaron.j.watkins@gmail.com> wrote:
> > > Hi there,
> > >
> > > I have been doing some stress testing of syslog-ng as a possible
> > > facilitator of event messaging for use on our network. I have set up a
> > > system as below:
> > >
> > > Event Process
> > > |
> > > \_/
> > > FIFO/pipe
> > > |
> > > _____\_/_______
> > > [ ]
> > > [ Machine 1 ]
> > > [ running ]
> > > [ syslog-ng ]
> > > [______________]
> > > |
> > > \_/
> > > TCP socket
> > > |
> > > _____\_/_______
> > > [ ]
> > > [ Machine 2 ]
> > > [ running ]
> > > [ syslog-ng ]
> > > [______________]
> > > |
> > > \_/
> > > FIFO/pipe
> > > |
> > > \_/
> > > cat
> > >
> > > Or to describe it, I have a process writing to a pipe on Machine 1,
> > > which syslog-ng then forwards via a socket to syslog-ng running on
> > > Machine 2, which then outputs to another pipe, which I am catting.
> > >
> > > My event process is in a while(true) loop, simply writing a number of
> > > messages to the pipe and then sleeping for 1 second.
> > >
> > > The problem I have discovered is that if I kill (-TERM or -9)
> > > syslog-ng running on Machine 2, some messages are lost before it
> > > starts buffering them in Machine 1. If I kill it while my event
> > > process is sleeping and then restart it, the first two messages (once
> > > the event process restarts transmitting) will be lost. Alternatively,
> > > if I kill it while my event process is writing data (and syslog-ng on
> > > Machine 1 is transmitting over the socket), normally about five
> > > messages will be lost.
> > >
> > > Is this a recognised problem and is there a way to work around this?
> > > Obviously, I would like to be able to get all of my messages through
> > > (up to the buffer limit) without having to reprocess the messages from
> > > the local event copy file.
> > >
> > > I am running this system on Solaris 5.8.
> > >
> > > The code for my event process is:
> > >
> > > #!/bin/sh
> > > while [ 1 ]
> > > do
> > > count=0
> > > while [ $count -le 40 ]
> > > do
> > > echo "${count}: Hello World" >> event_pipe
> > > count=`expr ${count} + 1`
> > > done
> > > sleep 1
> > > done
> > >
> > > My Machine 1 config file:
> > >
> > > options {
> > > sync(0); # Sends message as soon as it's received
> > > time_reopen(10); # Wait 10 before reopening closed connections
> > > log_fifo_size(500); # The number of messages that can go in the output queue
> > > create_dirs(yes); # Whether to create destination directories, if required
> > > };
> > >
> > > source src_internalEvents {
> > > internal();
> > > };
> > >
> > > source src_InputPipe {
> > > pipe( "/tmp/event_pipe" );
> > > };
> > >
> > > destination dst_localLog {
> > > file( "/data/gen/log/nievent_all.log"
> > > template("$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM):
> > > $MSG\n" )); };
> > >
> > > destination dst_localEventCopy {
> > > file( "/data/gen/log/nievent_copy.log"
> > > template("$MSG\n") );
> > > };
> > >
> > > destination dst_OutputSocket {
> > > tcp( "imgwrkfl-inp" port(1515) );
> > > };
> > >
> > > log {
> > > source( src_InputPipe );
> > > destination( dst_localLog );
> > > destination( dst_localEventCopy );
> > > destination( dst_OutputSocket );
> > > };
> > >
> > > log {
> > > source( src_internalEvents );
> > > source( src_InputPipe );
> > > destination( dst_localLog );
> > > flags( fallback );
> > > };
> > >
> > > For Machine 2:
> > >
> > > options {
> > > sync(0); # Sends message as soon as it's received
> > > time_reopen(10); # Wait 10 before reopening closed connections
> > > log_fifo_size(500); # The number of messages that can go in the output queue
> > > create_dirs(yes); # Whether to create destination directories, if required
> > > };
> > >
> > > source src_internalEvents {
> > > internal();
> > > };
> > >
> > > source src_InputSocket {
> > > tcp( port(1515) );
> > > };
> > >
> > > destination dst_localLog {
> > > file( "/data/gen/log/nievent_all.log"
> > > template( "$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): $MSG\n" ) );
> > > };
> > >
> > > destination dst_localEventCopy {
> > > file( "/data/gen/log/nievent_copy.log"
> > > template( "$MSG\n") );
> > > };
> > >
> > > destination dst_OutputPipe {
> > > pipe( "/tmp/prepress/nievents_pipe"
> > > template( "$MSG\n") );
> > > };
> > >
> > > log {
> > > source( src_InputSocket );
> > > destination( dst_localLog );
> > > destination( dst_localEventCopy );
> > > destination( dst_OutputPipe );
> > > };
> > >
> > > log {
> > > source( src_internalEvents );
> > > source( src_InputSocket );
> > > destination( dst_localLog );
> > > flags( fallback );
> > > };
> > >
> > > Thanks in advance for your help.
> > >
> > > Aaron Watkins
> > > _______________________________________________
> > > syslog-ng maillist - syslog-ng@lists.balabit.hu
> > > https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> > >
> > >
> >
>
>
> _______________________________________________
> syslog-ng maillist - syslog-ng@lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>
>