On Thu, 2008-05-29 at 11:07 -0700, Chris Wagner wrote:
We run syslog-ng (both 1.x and now 2.0.9) on FreeBSD writing log files to a NetApp NFS server. At a very low load we start dropping inbound packets (udp, large socket buffers). A quick tcpdump showed a pathological interaction with NFS revolving around how NFS (at least on BSD) deals with APPEND mode. The upshot is that for each write, it flushes the attributes and actually re-reads the partial file block back from the server!
ouch. that sounds bad.
A simple change to the source – to remove O_APPEND had great results – we can handle 10x the number of inbound messages.
Our .conf file has every logging host going to a separate file, and with a quick scan of the code it sure looked like syslog-ng won’t open the same file more than once anyway (I could be wrong here).
It depends on the configuration, if you configure the same file multiple times, syslog-ng will open them multiple times. O_APPEND ensures that it properly writes to the end of file all times. What do you think, would an lseek + write combination solve the issue? e.g. open without O_APPEND, and use lseek(fd, 0, SEEK_END) before calling write().
So, a couple questions (I scanned the last few months of archives but didn’t see any discussion about this) - what breaks if anything by turning off APPEND, is this interesting to others/how might I get this into future releases ( I can see this being a per-destination option).
Sure, I'm interested in performance corner cases like this. The only question is how to solve it without affecting others. -- Bazsi