[syslog-ng] syslog_ng and NFS

[Balazs Scheidler]

On Thu, 2008-05-29 at 11:07 -0700, Chris Wagner wrote:
> We run syslog-ng (both 1.x and now 2.0.9) on FreeBSD writing log files
> to a NetApp NFS server. At a very low load we start dropping inbound
> packets (udp, large socket buffers). A quick tcpdump showed a
> pathological interaction with NFS revolving around how NFS (at least
> on BSD) deals with APPEND mode. The upshot is that for each write, it
> flushes the attributes and actually re-reads the partial file block
> back from the server!

ouch. that sounds bad.

> 
> A simple change to the source – to remove O_APPEND had great results –
> we can handle 10x the number of inbound messages.
> 
> Our .conf file has every logging host going to a separate file, and
> with a quick scan of the code it sure looked like syslog-ng won’t open
> the same file more than once anyway (I could be wrong here).

It depends on the configuration, if you configure the same file multiple
times, syslog-ng will open them multiple times. O_APPEND ensures that it
properly writes to the end of file all times.

What do you think, would an lseek + write combination solve the issue?
e.g. open without O_APPEND, and use lseek(fd, 0, SEEK_END) before
calling write().

> So, a couple questions (I scanned the last few months of archives but
> didn’t see any discussion about this) - what breaks if anything by
> turning off APPEND, is this interesting to others/how might I get this
> into future releases ( I can see this being a per-destination option).

Sure, I'm interested in performance corner cases like this. The only
question is how to solve it without affecting others.

-- 
Bazsi