[syslog-ng] logline corruption

[Erik Williamson]

Hi All,

I just 'inherited' a large syslog-ng setup that involves on average
1.5 Terabytes of data generated per day.  Logs come from many hundreds
of hosts, to one of 10 or more syslog-ng servers (selected via DNS
round-robin).  From there, it goes straight to an NFS server, without
being written to a local disk.  There are many destination files on
the server, but it is possible that more than one logging server may
be attempting to write to the same destination file at the same time.

We're seeing corruption where a logline will have another spliced in
the middle, parts here, parts there, etc.  Not constantly, but enough
to give us a headache.

Stats on the NFS server show that the cpu can get maxed out, but the
netwok pipe is not full.
The individual logger servers show that syslog-ng is using about 10%
CPU on dual 3gHz P4's

I''m thinking that it's the nfs server that cannot cope with multiple
sources trying to write to the same file, but would like to throw this
out to the masses - do you have any hints as to how I could verify
where the source of the problem is?  I do realize that this setup is
less-than-optimal, and will be taking steps to repair this as time
goes on.  Alas, I must get corruption down first... then re-architect
second.

Any hints or suggestions would be greatly appreciated!

Thanks Very Much,
Erik.