[syslog-ng]losing messages using syslog-ng-1.4.7 & libol-0.2.20

[matthew.copeland@honeywell.com]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to mime@docserver.cac.washington.edu for more info.

---1768452832-505006337-972341286=:29132
Content-Type: TEXT/PLAIN; charset=US-ASCII


I am using syslog-ng 1.4.7 and libol 0.2.20 and am attempting to do some
tests to make sure that I can get the level of performance that I need.  I
am having the worst trouble though trying to isolate whether a problem is
occuring in either syslog-ng or within the networking code of the machines
that I am using.  I've managed to hack things in the network code for
some of the machines to get better performance for some of the tests, but
we have dropped down the requirements now.

I am sending along some test code that should compile on either red hat
6.2, red hat 7.0, and hopefully others.  (Haven't tried) The problem I am
running into is that I lose anywhere between 30 and  800 messages.  As the
message number goes up and number of clients, I lose more messages.

Some results I have gotten, with 100 threads connecting to the server, I
can send 60,000 messages in about 30 seconds without a problem (200
bytes), but when I increase the number to 300 threads, and I have it
sending for 30 seconds for a total of 36,000 messages, I start to lose
messages.  Notice in the first case I was sending more messages and had it
sleeping for less time between instances, in the second case it is
sleeping longer and sending less messages per second per thread.

I have tried it with multiple different machines and types of varying
speeds, and it happens on all of them.  For the latest run, I have for teh
sending machine an AMD athlon 1Ghz w 256M RAM, with an ATA 100 Hard drive
running on ATA66 running red hat 7.0.  For the server, I have a PII 300Mhz
with 180M RAM, a ATA33 hard drive running without acceleration.  The
server is running red hat 6.2.  I have tried the numbers with the sender
and receiver switched, but I lose more messages that way.

If you look in the code at logconf.h, you can change the #defines.  The
logtest.c just creates the threads.  log_thread.c has the send_messages
function that simulates a syslog-ng client.  If anyone can provide any
help, it would be greatly appreciated, since I have to be able to show
this working before I can get the full sign-off to use it for our the
applications we are building.


Matthew M. Copeland
Software Engineer
matthew.copeland@honeywell.com

---1768452832-505006337-972341286=:29132
Content-Type: APPLICATION/x-gzip; name="logtest.tgz"
Content-Transfer-Encoding: BASE64
Content-ID: <Pine.LNX.4.21.0010232248060.29132@fisb.gaa.aro.allied.com>
Content-Description: 
Content-Disposition: attachment; filename="logtest.tgz"

H4sIAAu99DkAA+1aeZPauBLPv8OnUM1usjBmYhswhkwmVRzmGsDc174UZWyB
zSGDj+HY2u/+JLDBDExSr3aSbNXzrxIM3VKr3Wq1ujuZ6xMLmhb97geCYWIM
z3H4yTB8PH548tH908E7hmd5jo3Go3s6/sq+A9yPVMqFbVqSAcDPWOrfiLmz
//g5tFQDSspH+a3XYFiGicdir+0/z8Q4d/+jDIfpLMvHmXeAeWtFruH/fP9/
05A8txUIPptbk7a2S2h+VL8EzsmmLs+gdU5H0NLwX1pDL8Zbiqafk2ykYeo5
zdIW8Ixy6/FA9facLutofEb8vHRHvlBJGb3UxtDQ5DV1As+6pgATImW4gKYp
TaAZ3JPuwNIyQiDwVwBgYCG2bAFVNy2ILHCnLh+8dGIdSVGMoYYAkhbwwNTw
yDlEE0t9ZB4CNE1osopdba6jycge//kVPILbprlLwK280HJROSsi1aywsY0x
VhNxtW7NqnK5UOuhImO2tOpaiaB+hFWNvtVJrLZ6dmOWI6sav2QFVCrL3fbs
KdF/RjWx125LC2HaGgiROdOoMKW62q20Ua7Nd8SOllCWtgyj6Ww5soiXGb6W
SsJyfvu0oiujFaWWSnTL6HVsExXaSSpT56fTFpQYJteTx5xYnXbpXrNnFMoF
CkamowbF96KokNzN6B3KxvuxXZbjSzv9KWUsKlRKnE8kGSqjdvYJKsXnRI7V
nioDkdMjkibmdWMdS9qZZdOgs5FKL7EzrVqmOp7rQs4obmbtVCzXU/WutMhU
yrPE4HlhTVGs2HzKFLMxy5pu6Ua/KykT1KJ0sZsZj5Ti1KTsUZwqC70dL9dy
nGnKSCywi6ZsqpEqU0ukOk/QKK0kGpVG+badVDu6toLFLjdfb9h5pBZl8iuZ
RwlRr6PotFi305mkQgucUOhu2TE9om1hnMkNIkbOmFuUFZ2sYzS9FbmM1Wcm
m+IO2ZRMiRWVij5PSo2oISuZ50kOxjPFWnoTU2M0I04Eq6B20lyO0tq6NFiJ
u+i2Py0P2rkpqrWWUz7OTVsTpTRK5BJwSUmtJr8bz21Ts7fF+HZc5LMaxRVX
vZoIn7vpnlpH1AwZtSdrRWnT2qou9hurRHVCGVIZ6RGz3zWq6wan2+nKopQc
aK0B1YlqqsAk4o1UcpWqzbadomEP2nFehi1eMDr8dFtrrLoVIzvt1591lCsm
8ju9t2YtcwHpVLbOzlfpjLKJZBt5ftDcdqpim5/ahV1jbW3WSWGhME1BFAcr
rldnNk2xmNDNLosy68SosN3OesxoK6Xl+hOUdr2cnVAS41qpk4v3ZpVOXInz
ibSM1PKgyjbzzS5rFGbdGZ2N5mC0HofJpCqr2qa9qVCV/LJgNDYNfp4o8LqU
svINTmnxbQo2IwLSi/yOLq8YYdk1I8rCSpX5ZXu7mdKtbTI7LtX47KZuseqg
L9GN6aRKVcbrUr4zNdq1zrhWbCakqm4XaLlWz0SXCp3azhLSgprUduIIUaXs
0ybKbXdTBCcps2JLulDuWdN+cmXFpYixXc4q41FsqsT6zLOW03pcph4XV0qe
76ert4fYcBkHGDYSjXH47ZN4SOA0iAyoCM1mKi8Mm8WB8PUUXDDrEF/wfO+Q
0xJ3QMZhzLOkE+TOJAIKRGIesYfAOkT2YgQNLPouGMTku9A+Ih6G2cjUJggq
+1c4hE88/BhC8aTg2RAcSIGDYLApZMRqtjlsicNWI1VtVootHG7ZQyoQAjRo
uyPSQqsrCNWho20T4OWvrC/rNrKgQaIsYZKLZYiDMr7VrSH5cUYeazj4qx76
/pX3PH1oziE8D+3WAtvQNgwc9l/MOVyJQ3wzIShbmo4eMMPdOWwqbAS8QYfx
a1Wbw+Bpv77g1w2Bv45GwYshebkNknlh1y/CZNDDccxeJvWIiSfaSeL9ifF3
IHAh8yjyOMM1pTYOBtUlVnYCLXK/jbbkBgs2hIrYwv4hNDpCIxQCj4+g2i6X
nY08ab6EhqEbwVsyFSAd2xdvhnLr0RtuNCvIhs51e2k7vP6BFkzlhsWq0Ao3
xczTsNlqCKlK2DUD1vVy5mfAvKpVTsJ2V4ClAxn7tAWdRQD4hoLkc/+xgAsT
K/SB2CMMmDAwtR3UsQoXl37INSYZ+tHU0HAsLbT5Fr+V8zoP5+ylbliYqVo6
MoOno2KqhI6dK+SYvyY2Wo5iWBuymx+OMsjaYaAu77+o++/DOU5q/mS+urTj
NpPZxzjx+juQqONa2THvpbXD4OVUgAODYyJnxW/uSEa35wr6wwKOTLI3Blzo
eG+IC33PcQKub7tnG78SeQT3zukEOs9xxey5Lkvz/ZgPp2khb0xwzj0eG2fA
/dn8+y/WYmhC+eHkFkucT1rj4O0hTIJDmPwEwHvFqxchmOA/6DZ8Fk7DQL5Q
xXWevRLBM5VCD99+3T13jO3qhEDMZh7ceAg+P16G5geHSVEXW3TKV4mfOK/p
zAuD2/fKPX7F9+Zt+PyGCJPbIeSIDeP44tlCcmCxBlf86Cj3sCI+4h63OdfL
6z5EmtdHLvzkdIYJ7INNX71Qzp3LczVcs/M3Nt6773jbidHdH8QFvOq+cAcv
69I1rvI9aoY8UdxUbUvR1+iKsSN744JXD6U7FehjcJp2e7TO34FfXaT6+GG4
0v9R33qN7/R/MDN66v/w/KH/E/P7Pz8DbluE9G3m2uhbXZE7pwT41Sr7eEN4
zj95vn3z9913zz8bIWf+cP65aIQh5z8aZf3z/zPwlv3Xa33eYzt4H1QWkoaC
pPQ9pFRONnRsOuCC689qu5IWGkMxN2wVcPmXbXp6Ey9KfULa5zDApbgSJcsy
sFjyIIUNfXfBxFkULu7I19CVqVgRBVqSrGLnsOBhXBjUDhoNM/gTl2clsVhN
pcvCKwI0pEJDs0xZhcpLCcVqQWgUW8NmpiBkQw/gjg5cyeYZ4MnmwYVhriTz
p+rpYJajUocCOPgh6LG0M/3rizzTC5IAh19jHi6F0IezmyIMnA46rgudBbB9
bpxUHAcC8gvnq3sFiSfc3OA/birqJNd7bUl1vyeTTPTmbycZvXEEJYggPzP9
x/DE/8Mp/wFrfC//i8aP//7HsbHIPv4zUT/+/wwEflMgLikhOGv5gduxZo5u
XzJJQwokk8kj/ayRjHf5yHi9iRvhyJYfB15rB0dP7Iugh5nMPqBbKsQ1uT4x
pAVYa/M5rt8ntgGBbpN/qVvjqwZtwbElben7hPYj2IfaX230fxHc81+RZtji
c/gj1vhe/sdwcef8R+JcdJ//kTDgn/+fgEzmcSLLgUyunMo3H8H9BNx3JXya
7mVwX/wYKGddRvkjuJ876QQIiOlS89GtGXTgyR31QK2Rr5AOGmZh+kRDk0AA
i/x083uQsEJ7jmJoz9AIBBwaYRKZOJPB3zKZELjXgTveZf0edNQJkU7wScqn
G+/6HvFYmZO0Exm8Pjzg/fXpxvNL9s5Sj2riT0cl71is3tE4Z9qp4HjRgmPJ
dSnrxAp4df0EviHr+N93roo7cQPyHEroU+DGWIB70vHDQdO1sLOu11Zn5vnV
3urjreHGf6/3vvUa38v/cOQ/xv8IR/K/CBPz87+fgv+p/g+QivtYw994SnIA
HllcpZ03Da9Wgga0bAMB1q/efPjw4cOHDx8+fPjw4cOHDx8+fPjw4cOHDx8+
fPjw4cOHDx8+fPjw4eMf4b+Us1ExAFAAAA==
---1768452832-505006337-972341286=:29132--