Hi, I discovered a problem recently and I wanted to let you know about it. The Linux kernel may drop bytes from its ringbuffer which causes a garbled message to reach the logs. For example, I enabled logging on my notebook for all loopback traffic, and used ping -f to generate a lot of messages. To avoid syslog-ng's complexity in the picture, I simply used "cat" to read kernel messages: # cat /proc/kmsg > kmsg ## generated about 40000 packets # grep -v '^<4>IN=lo OU' kmsg RC=127.0.0.1 DST=127.0.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=15957 DF PROTO=ICMP TYPE=8 CODE=0 ID=21848 SEQ=21822 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=11274 DF PROTO=ICMP TYPE=8 CODE=0 ID=22104 SEQ=2604 As you can see both messages missed the syslog header (the string <4>), but messages might be garbled in different ways, in general the linux kernel shifts the ring buffer whenever it is full, without flow control. I also checked the the cat process with "strace" to confirm that the kernel itself returns garbled messages. A workaround is to increase the kernel ring buffer size which is possible using a compile time option in recent kernels, but it is also possible to change by patching the value of LOG_BUF_LEN in the beginning kernel/printk.c to a greater value (must be power of two) However this will not necessarily fix the problem, the kernel can always "outperform" its userspace, e.g. it can generate way more messages than a userspace program can handle. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
participants (1)
-
Balazs Scheidler