October 2004 - syslog-ng - lists.balabit.hu

Re: [syslog-ng]syslog-ng loses messages
by Aaron watkins 11 Oct '04

11 Oct '04

Dave, > Dave Johnson syslog-ng(a)lists.balabit.hu > Thu, 7 Oct 2004 10:47:14 -0500 > > We have a couple solaris boxes that do logging with pipes and we log a > large amount of data, thats why I'm interested in these tests. I can > see if I can setup a quick testbed next week. > > ********** > > Otherwise, I'm wondering if the packets are throttled, what happens.. IE: > > --------------------------- > cat > /tmp/smallsleep.c <<ENDHERE > #include <unistd.h> > > main () { > usleep(50000); /* 20 hz */ > } > ENDHERE > gcc -o /tmp/smallsleep /tmp/smallsleep.c > ----------------------------- > in the do -> done segment of your for loop add `/tmp/smallsleep` > > How does this impact lost messages? > In the situation where the receiving server is killed while the sending server has stopped sending messages (ie. during the sleep 1 command), there is no affect (ie. Message 0 and 1 are still lost). In the situation where the receiving server is killed while the sending server is still sending messages, the throttling you suggested reduces the number of messages that are lost to 2 (similar to the first test). Test 1: Message 12 and 13 lost Test 2: Message 25 and 26 lost It's an improvement, but unfortunately, its still not quite reliable enough and when used in practice, I'm unsure if our events (which will be asynchronous by their very nature) could possibly be throttled... Do you have any more ideas??? Thanks, Aaron Watkins

2 1

tcp destination unavailable ... what happens ?
by Ali EL MOUSSAWI 11 Oct '04

11 Oct '04

Hi all, Could you please tell me what can happen if the destination tcp driver is unavailable ? I made the test with a wrong IP address in destination on my machine running syslog-ng, my software generates messages every minutes, and these messages are redirected to the destination tcp driver. The only think I saw is that in /var/adm/messages logs, the following message was repeated every 4 minutes or so ... Oct 11 10:26:39 met-ixb syslog-ng[216]: Error connecting to remote host AF_INET(192.168.192.168:514), reattempting in 10 seconds Is there any risk for my machine if the destination tcp driver is no longer available ? Will there be any buffer over size or something like that ? Or ressources over load ? Thanks a lot Ali

1 0

Re: [syslog-ng]syslog-ng loses messages
by Aaron watkins 07 Oct '04

07 Oct '04

Dave, > * What are the numbers of lines that get dropped in each of those tests? If I kill (-TERM or -9) the receiving server while messages are not being written to the input pipe, I will lose message 0 and message 1. When I kill -TERM while writing to the input pipe: Test 1: Messages 27->34 are lost (8 in total) Test 2: Messages 24->31 are lost (8 in total) When I kill -9 while writing to the input pipe: Test 1: Messages 35->38 are lost (4 in total) Test 2: Messages 23->29 are lost (7 in total) > * What happens in your test if you change machine2 fifo's to 0? I presume you mean the log_fifo_size option... When I set this to 0, the following messages got through: Iteration 1: Message 0,1,14,28 Iteration 2: Message 0,1,12,21,32 Iteration 3: Message 0,1,12,21,30,39 That is to say, in every iteration, around 37 messages were lost. And this was with the system operating normally (ie. without killing the machine2's syslog-ng). Any ideas? Thanks, Aaron > > On Fri, 1 Oct 2004 12:38:56 +0100, Aaron watkins > <aaron.j.watkins(a)gmail.com> wrote: > > Hi there, > > > > I have been doing some stress testing of syslog-ng as a possible > > facilitator of event messaging for use on our network. I have set up a > > system as below: > > > > Event Process > > | > > \_/ > > FIFO/pipe > > | > > _____\_/_______ > > [ ] > > [ Machine 1 ] > > [ running ] > > [ syslog-ng ] > > [______________] > > | > > \_/ > > TCP socket > > | > > _____\_/_______ > > [ ] > > [ Machine 2 ] > > [ running ] > > [ syslog-ng ] > > [______________] > > | > > \_/ > > FIFO/pipe > > | > > \_/ > > cat > > > > Or to describe it, I have a process writing to a pipe on Machine 1, > > which syslog-ng then forwards via a socket to syslog-ng running on > > Machine 2, which then outputs to another pipe, which I am catting. > > > > My event process is in a while(true) loop, simply writing a number of > > messages to the pipe and then sleeping for 1 second. > > > > The problem I have discovered is that if I kill (-TERM or -9) > > syslog-ng running on Machine 2, some messages are lost before it > > starts buffering them in Machine 1. If I kill it while my event > > process is sleeping and then restart it, the first two messages (once > > the event process restarts transmitting) will be lost. Alternatively, > > if I kill it while my event process is writing data (and syslog-ng on > > Machine 1 is transmitting over the socket), normally about five > > messages will be lost. > > > > Is this a recognised problem and is there a way to work around this? > > Obviously, I would like to be able to get all of my messages through > > (up to the buffer limit) without having to reprocess the messages from > > the local event copy file. > > > > I am running this system on Solaris 5.8. > > > > The code for my event process is: > > > > #!/bin/sh > > while [ 1 ] > > do > > count=0 > > while [ $count -le 40 ] > > do > > echo "${count}: Hello World" >> event_pipe > > count=`expr ${count} + 1` > > done > > sleep 1 > > done > > > > My Machine 1 config file: > > > > options { > > sync(0); # Sends message as soon as it's received > > time_reopen(10); # Wait 10 before reopening closed connections > > log_fifo_size(500); # The number of messages that can go in the output queue > > create_dirs(yes); # Whether to create destination directories, if required > > }; > > > > source src_internalEvents { > > internal(); > > }; > > > > source src_InputPipe { > > pipe( "/tmp/event_pipe" ); > > }; > > > > destination dst_localLog { > > file( "/data/gen/log/nievent_all.log" > > template("$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): > > $MSG\n" )); }; > > > > destination dst_localEventCopy { > > file( "/data/gen/log/nievent_copy.log" > > template("$MSG\n") ); > > }; > > > > destination dst_OutputSocket { > > tcp( "imgwrkfl-inp" port(1515) ); > > }; > > > > log { > > source( src_InputPipe ); > > destination( dst_localLog ); > > destination( dst_localEventCopy ); > > destination( dst_OutputSocket ); > > }; > > > > log { > > source( src_internalEvents ); > > source( src_InputPipe ); > > destination( dst_localLog ); > > flags( fallback ); > > }; > > > > For Machine 2: > > > > options { > > sync(0); # Sends message as soon as it's received > > time_reopen(10); # Wait 10 before reopening closed connections > > log_fifo_size(500); # The number of messages that can go in the output queue > > create_dirs(yes); # Whether to create destination directories, if required > > }; > > > > source src_internalEvents { > > internal(); > > }; > > > > source src_InputSocket { > > tcp( port(1515) ); > > }; > > > > destination dst_localLog { > > file( "/data/gen/log/nievent_all.log" > > template( "$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): $MSG\n" ) ); > > }; > > > > destination dst_localEventCopy { > > file( "/data/gen/log/nievent_copy.log" > > template( "$MSG\n") ); > > }; > > > > destination dst_OutputPipe { > > pipe( "/tmp/prepress/nievents_pipe" > > template( "$MSG\n") ); > > }; > > > > log { > > source( src_InputSocket ); > > destination( dst_localLog ); > > destination( dst_localEventCopy ); > > destination( dst_OutputPipe ); > > }; > > > > log { > > source( src_internalEvents ); > > source( src_InputSocket ); > > destination( dst_localLog ); > > flags( fallback ); > > }; > > > > Thanks in advance for your help. > > > > Aaron Watkins > > _______________________________________________ > > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > > > > >

2 1

perl syslog::parse problem
by lists＠mhcsoftware.de 07 Oct '04

07 Oct '04

hi, i've switched to syslog-ng and now mailgraph stoped working. i've read that there is a patch requiered for syslog::parse but i can't find it anywhere. can some one please point me to this patch ? TIA matthias -- Matthias Henze matthias(a)mhcsoftware.de Use PGP!! http://www.mhcsoftware.de/MatthiasHenze.asc - - - - - - - - - - - - - - - - - - - - - - - - - - - - MHC SoftWare GmbH voice: +49-(0)9533-92006-0 Fichtera 17 fax: +49-(0)9533-92006-6 96274 Itzgrund/Germany e-Mail: info(a)mhcsoftware.de - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 0

syslog-ng loses messages
by Aaron watkins 01 Oct '04

01 Oct '04

Hi there, I have been doing some stress testing of syslog-ng as a possible facilitator of event messaging for use on our network. I have set up a system as below: Event Process | \_/ FIFO/pipe | _____\_/_______ [ ] [ Machine 1 ] [ running ] [ syslog-ng ] [______________] | \_/ TCP socket | _____\_/_______ [ ] [ Machine 2 ] [ running ] [ syslog-ng ] [______________] | \_/ FIFO/pipe | \_/ cat Or to describe it, I have a process writing to a pipe on Machine 1, which syslog-ng then forwards via a socket to syslog-ng running on Machine 2, which then outputs to another pipe, which I am catting. My event process is in a while(true) loop, simply writing a number of messages to the pipe and then sleeping for 1 second. The problem I have discovered is that if I kill (-TERM or -9) syslog-ng running on Machine 2, some messages are lost before it starts buffering them in Machine 1. If I kill it while my event process is sleeping and then restart it, the first two messages (once the event process restarts transmitting) will be lost. Alternatively, if I kill it while my event process is writing data (and syslog-ng on Machine 1 is transmitting over the socket), normally about five messages will be lost. Is this a recognised problem and is there a way to work around this? Obviously, I would like to be able to get all of my messages through (up to the buffer limit) without having to reprocess the messages from the local event copy file. I am running this system on Solaris 5.8. The code for my event process is: #!/bin/sh while [ 1 ] do count=0 while [ $count -le 40 ] do echo "${count}: Hello World" >> event_pipe count=`expr ${count} + 1` done sleep 1 done My Machine 1 config file: options { sync(0); # Sends message as soon as it's received time_reopen(10); # Wait 10 before reopening closed connections log_fifo_size(500); # The number of messages that can go in the output queue create_dirs(yes); # Whether to create destination directories, if required }; source src_internalEvents { internal(); }; source src_InputPipe { pipe( "/tmp/event_pipe" ); }; destination dst_localLog { file( "/data/gen/log/nievent_all.log" template("$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): $MSG\n" )); }; destination dst_localEventCopy { file( "/data/gen/log/nievent_copy.log" template("$MSG\n") ); }; destination dst_OutputSocket { tcp( "imgwrkfl-inp" port(1515) ); }; log { source( src_InputPipe ); destination( dst_localLog ); destination( dst_localEventCopy ); destination( dst_OutputSocket ); }; log { source( src_internalEvents ); source( src_InputPipe ); destination( dst_localLog ); flags( fallback ); }; For Machine 2: options { sync(0); # Sends message as soon as it's received time_reopen(10); # Wait 10 before reopening closed connections log_fifo_size(500); # The number of messages that can go in the output queue create_dirs(yes); # Whether to create destination directories, if required }; source src_internalEvents { internal(); }; source src_InputSocket { tcp( port(1515) ); }; destination dst_localLog { file( "/data/gen/log/nievent_all.log" template( "$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): $MSG\n" ) ); }; destination dst_localEventCopy { file( "/data/gen/log/nievent_copy.log" template( "$MSG\n") ); }; destination dst_OutputPipe { pipe( "/tmp/prepress/nievents_pipe" template( "$MSG\n") ); }; log { source( src_InputSocket ); destination( dst_localLog ); destination( dst_localEventCopy ); destination( dst_OutputPipe ); }; log { source( src_internalEvents ); source( src_InputSocket ); destination( dst_localLog ); flags( fallback ); }; Thanks in advance for your help. Aaron Watkins

2 1

syslog-ng to MySQL question
by Ron Clark 01 Oct '04

01 Oct '04

Hello all, This may be off topic, but I would be very appreciative of any help. I am using the sqlsyslogd file from http://www.frasunek.com/sources/security/sqlsyslogd/. But when it reads from the logs coming into syslog, on the time stamp field, it adds some text in <> signs, and does not fully capture the time stamp. Does anyone know what these are? And more important, how does one get the script to get fully imput the time stamp in the syslogs? Does anyone have any tricks to the sqlsyslog.c file that might fix this? If this is not the correct forum, then please forward me to the right person or place. Thank you in advance, Ron Clark actual syslog: Sep 30 16:03:05 system1 /kernel: mysql entry: | 15468 | <118>Sep 30 18: | system1 | psmintr |

2 1

/proc/kmsg and chrooted syslog-ng
by Michael D. (Mick) Bauer 01 Oct '04

01 Oct '04

One more ignorance-betraying question about Syslog-ng chroot-jails: what should one do about /proc/kmsg? Maybe "touch /var/logjail/proc/kmsg; ln -s /var/logjail/proc/kmsg /proc/kmsg"? And do I take it that, per another recent posting, one should similarly create a new dev/log in the jail and link to it from /dev/log? I'd like to post a revised Syslog-ng chroot jail procedure later this week. Thanks to the two of you who've replied so far! Mick /-------------------------------------------------\ | Michael D. (Mick) Bauer | | Security Editor, Linux Journal | | Dir. of Value-Subtracted Svcs., Wiremonkeys.org | \-------------------------------------------------/

3 2

syslog-ng to MySQL question
by Ron Clark 01 Oct '04

01 Oct '04

Hello all, This may be off topic, but I would be very appreciative of any help. I am using the sqlsyslogd file from http://www.frasunek.com/sources/security/sqlsyslogd/. But when it reads from the logs coming into syslog, on the time stamp field, it adds some text in <> signs, and does not fully capture the time stamp. Does anyone know what these are? And more important, how does one get the script to get fully imput the time stamp in the syslogs? Does anyone have any tricks to the sqlsyslog.c file that might fix this? If this is not the correct forum, then please forward me to the right person or place. Thank you in advance, Ron Clark actual syslog: Sep 30 16:03:05 system1 /kernel: mysql entry: | 15468 | <118>Sep 30 18: | system1 | psmintr |

1 0