[syslog-ng]garbled kernel messages

[Balazs Scheidler]

Hi,

I discovered a problem recently and I wanted to let you know about it.
The Linux kernel may drop bytes from its ringbuffer which causes a
garbled message to reach the logs.

For example, I enabled logging on my notebook for all loopback traffic,
and used ping -f to generate a lot of messages. To avoid syslog-ng's
complexity in the picture, I simply used "cat" to read kernel messages:

# cat /proc/kmsg > kmsg
## generated about 40000 packets
# grep -v '^<4>IN=lo OU' kmsg
RC=127.0.0.1 DST=127.0.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=15957 DF PROTO=ICMP TYPE=8 CODE=0 ID=21848 SEQ=21822
IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=11274 DF PROTO=ICMP TYPE=8 CODE=0 ID=22104 SEQ=2604

As you can see both messages missed the syslog header (the string <4>), but messages
might be garbled in different ways, in general the linux kernel shifts the ring
buffer whenever it is full, without flow control.

I also checked the the cat process with "strace" to confirm that the kernel itself 
returns  garbled messages.

A workaround is to increase the kernel ring buffer size which is possible using a 
compile time option in recent kernels, but it is also possible to change
by patching the value of LOG_BUF_LEN in the beginning kernel/printk.c to a greater
value (must be power of two)

However this will not necessarily fix the problem, the kernel can always "outperform" 
its userspace, e.g. it can generate way more messages than a userspace program 
can handle.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1