[syslog-ng] On FreeBSd: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Traiano Welcome]

Hi List

I've been seeing the following in the messages log of my freebsd syslog
server for quite some time now:

---
Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while
writing; fd='12', error='No buffer space available (55)'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken;
time_reopen='60'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while
writing; fd='13', error='No buffer space available (55)'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken;
time_reopen='60'
---

These happen at a frequency of about 7 per minute on average. See attached
trend graphs for an idea of the volume of traffic we're doing, as well as
the memory and cpu utilisation trends on this server during this period.
As can be seen from the graphs, load does not seem to be the issue.
Occasionally during the week, the system freezes and requires a reboot, I
think it's related to the above message, though I'm not sure.

My question is: What does this error mean, and how can I resolve it?

I have tried to frame this as an operating system kernel resource issue,
and experimented with increasing the freebsd kernel sysctls for UDP
performance:

---
[root at syslog2 <mailto:root at syslog2.ops.mtnbusiness.co.za> /var/log]#
sysctl kern.ipc.nmbclusters=102400
kern.ipc.nmbclusters: 25600 -> 102400

[root at syslog2 <mailto:root at syslog2.ops.mtnbusiness.co.za> /var/log]#
sysctl kern.ipc.maxsockbuf=201326592
kern.ipc.maxsockbuf: 100663296 -> 201326592

[root at syslog2 <mailto:root at syslog2.ops.mtnbusiness.co.za> /var/log]#
sysctl net.inet.udp.recvspace=33554432
net.inet.udp.recvspace: 16777216 -> 33554432
---

This has reduced the frequency of the errors a little, but in general the
problem still remains.

Syslog version:

--
 [root at syslog2]# syslog-ng -V
 syslog-ng 2.0.10

--

FreeBSD version:

--
FreeBSD 7.2-STABLE #0
--

Any help would be much appreciated!
Traiano
 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: memory-week.png
Type: image/png
Size: 22863 bytes
Desc: memory-week.png
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120320/1abf8c84/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: if_bce0-day.png
Type: image/png
Size: 23391 bytes
Desc: if_bce0-day.png
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120320/1abf8c84/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cpu-day.png
Type: image/png
Size: 18106 bytes
Desc: cpu-day.png
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120320/1abf8c84/attachment-0005.png