[syslog-ng]syslog-ng loses messages

[Dave Johnson]

The two questions are:
1) where in the process do messages 0,1 get lost
2) where in the process do messages x get lost when high load

In theory, since using TCP this shouldn't be the problem... ?

------------

[system 1]
custom.program ->
pipe ->
syslog-ng

-> TCP / [network] ->

[system 2]
syslog-ng ->
pipe ->
listening process.

*--------------------------*
* Can we try using "logger" as opposed to "event_pipe" in the while loop?
* Can we `snoop -p -o /tmp/outout port 1515` on server2 and see if the
packets get sent?
* If that doesn't work, can we try without the templates?
* Where are the messages lost?  system1 local files? system2 local
files?  system2 pipe output?
* Solarisms?  nscd running?  kernel-revs?

----

Anybody else have any ideas? 


On Fri, 8 Oct 2004 12:22:45 +0100, Aaron watkins
<aaron.j.watkins@gmail.com> wrote:
> Dave,
> 
> > Dave Johnson syslog-ng@lists.balabit.hu
> > Thu, 7 Oct 2004 10:47:14 -0500
> >
> > We have a couple solaris boxes that do logging with pipes and we log a
> > large amount of data, thats why I'm interested in these tests.  I can
> > see if I can setup a quick testbed next week.
> >
> > **********
> >
> > Otherwise, I'm wondering if the packets are throttled, what happens..  IE:
> >
> > ---------------------------
> > cat > /tmp/smallsleep.c <<ENDHERE
> > #include <unistd.h>
> >
> > main () {
> > usleep(50000);  /* 20 hz */
> > }
> > ENDHERE
> > gcc -o /tmp/smallsleep /tmp/smallsleep.c
> > -----------------------------
> > in the do -> done segment of your for loop add `/tmp/smallsleep`
> >
> > How does this impact lost messages?
> >
> 
> In the situation where the receiving server is killed while the
> sending server has stopped sending messages (ie. during the sleep 1
> command), there is no affect (ie. Message 0 and 1 are still lost).
> 
> In the situation where the receiving server is killed while the
> sending server is still sending messages, the throttling you suggested
> reduces the number of messages that are lost to 2 (similar to the
> first test).
> 
> Test 1: Message 12 and 13 lost
> Test 2: Message 25 and 26 lost
> 
> It's an improvement, but unfortunately, its still not quite reliable
> enough and when used in practice, I'm unsure if our events (which will
> be asynchronous by their very nature) could possibly be throttled...
> 
> Do you have any more ideas???
> 
> Thanks,
> 
> 
> Aaron Watkins
> _______________________________________________
> syslog-ng maillist  -  syslog-ng@lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
>