[syslog-ng] Messages starting with tab not processed cleanly

[Jon Nathan]

We're running into an issue where some syslog messages on our
centralized syslog-ng server lack originating hostnames.  These
messages happen to be formatted funny - the message part starts
with a tab (\0x09) character.  Solaris 10 x86 everywhere, Syslog-ng
3.1.4, Eventlog 0.2.12.

Client syslog:

Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.warning] WARNING: /pci at 7b,0/pci1022,7458 at 11/pci1000,3060 at 2/sd at 0,0 (sd2):
Sep 15 20:30:21 h.example.com         Error for Command: read                    Error Level: Fatal
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   Requested Block: 16065                     Error Block: 16065
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   Vendor: SEAGATE                            Serial Number: 074093HVSX
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   Sense Key: Aborted Command
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   ASC: 0x8 (LUN communication time-out), ASCQ: 0x1, FRU: 0x81


This shows up in my centralized syslog-ng server as:

Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.warning] WARNING: /pci at 7b,0/pci1022,7458 at 11/pci1000,3060 at 2/sd at 0,0 (sd2):
Sep 15 20:30:21         Error for Command: read                    Error Level: Fatal
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   Requested Block: 16065                     Error Block: 16065
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   Vendor: SEAGATE                            Serial Number: 074093HVSX
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   Sense Key: Aborted Command
Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice]   ASC: 0x8 (LUN communication time-out), ASCQ: 0x1, FRU: 0x81

Note that the 2nd line doesn't have a hostname.  

This is the message coming from the host (tcpdump -n -v -s 0 "port 514"):

20:30:21.454902 IP (tos 0x0, ttl 255, id 43670, offset 0, flags [DF], proto UDP (17), length 141)
    10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 113
        Facility kernel (0), Severity warning (4)
        Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.warning] WARNING: /pci at 7b,0/pci1022,7458 at 11/pci1000,3060 at 2/sd at 0,0 (sd2):
20:30:21.454941 IP (tos 0x0, ttl 255, id 43671, offset 0, flags [DF], proto UDP (17), length 109)
    10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 81
        Facility kernel (0), Severity warning (4)
        Msg: Sep 15 20:30:21 \0x09Error for Command: read                    Error Level: Fatal
20:30:21.454966 IP (tos 0x0, ttl 255, id 43672, offset 0, flags [DF], proto UDP (17), length 139)
    10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 111
        Facility kernel (0), Severity notice (5)
        Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09Requested Block: 16065                     Error Block: 16065
20:30:21.454986 IP (tos 0x0, ttl 255, id 43673, offset 0, flags [DF], proto UDP (17), length 148)
    10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 120
        Facility kernel (0), Severity notice (5)
        Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09Vendor: SEAGATE                            Serial Number: 074093HVSX
20:30:21.455010 IP (tos 0x0, ttl 255, id 43674, offset 0, flags [DF], proto UDP (17), length 104)
    10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 76
        Facility kernel (0), Severity notice (5)
        Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09Sense Key: Aborted Command
20:30:21.455031 IP (tos 0x0, ttl 255, id 43675, offset 0, flags [DF], proto UDP (17), length 137)
    10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 109
        Facility kernel (0), Severity notice (5)
        Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09ASC: 0x8 (LUN communication time-out), ASCQ: 0x1, FRU: 0x81

(Hostnames/IPs sanitized)

The second message starts with a tab (\0x09) after the timestamp.
This seems to throw off syslog-ng such that it won't append a
hostname to the message.  We do a lot of parsing that relies on the
hostname being present in all messages.  Any idea why this seems
to break syslog-ng, or how to fix it?

Our syslog-ng.conf is relatively generic.  Here are the relevant parts:

filter f_syslog {
        level(info..emerg);
};

destination d_syslog {
        file("/syslog/logs/$YEAR/$MONTH/$DAY/syslog" flags(no-multi-line));
};

log{
        source(s_sys);
        source(s_remote);
        filter(f_syslog);
        destination(d_syslog);
};

(Sorry if this hits the list twice, I don't think my first try worked.)

Thanks,
-Jon