[syslog-ng] [Bug 215] syslog-ng v3 - tcp() does not support no-multi-line as docs reference.

[bugzilla at bugzilla.balabit.com]

https://bugzilla.balabit.com/show_bug.cgi?id=215





--- Comment #2 from Mike w <balabit at 32ths.com>  2013-01-03 15:51:28 ---
Hello Balazs,

Thanks for the thorough response and explanations, I responded inline.

> (In reply to comment #0)
> > So after doing some test cases, I've noticed that on no-multi-line is not stripping newlines on v3.3.7. I've not tested previous versions yet.
> > 
> no-multi-line is a transport independent flag and it is applied to the message as 
> decoded by the underlying transport protocol. The udp transport merely encloses 
> messages into their own UDP frame, embedded newlines are thus possible, and 
> no-multi-line removes them by replacing them with a space.
> 
> > When using the udp() source, it works just fine.
> > 
> > 
> > My test case:
> > 
> > # cat blah
> > line1
> > line2
> > line3
> > 
> > # cat blah | nc -r -n myhost 514
> > 
> > And on "myhost"
> > Jan  2 12:25:16 192.2.0.1 line1
> > Jan  2 12:25:16 192.2.0.1 line2
> > Jan  2 12:25:16 192.2.0.1 line3
> 
> this is the expected behaviour. If you want embedded newlines in messages you need to use a
> transport that supports those. Such is rfc5424 style messages used by the syslog driver.

Yes this makes sense now.

> What do you want to accomplish exactly?

I was hoping to accept windows logs from a snare universal forwarder from multiple devices. It sends multi-line windows logs but can only do it over TCP that
is not rfc5424 as far as I can tell and I wanted these messages to be reduced to one line.

> > 
> > So I assume this has something to do with it:
> > 
> > https://bugzilla.balabit.com/show_bug.cgi?id=74
> > 
> > Particularly about newlines being message terminators over tcp(), 
> 
> exactly.
> 
> > Maybe there is a way to adjust the terminator if this is the case? 
> 
> it's not possible right now, I'm afraid. 

This is what I was really needing. The terminator in my case would need to be \n\n since this is the true EOM with these logs and syslog-ng could then
determine the end of a message in the logs.
As it stands, it looks like to get this working on the syslog-ng side I would need to use program(), find true EOM (\n\n), strip the single \n and then return
the message as I want it.

> > I must admit, I was curious what syslog-ng would do if the message size exceeded the mtu of
> > the network. With UDP, I assume the end of the packet is the end of the message so any messages large enough to span two udp packets would actually be two log
> > entries. 
> 
> This is not true. IP level fragmentation happens below the UDP level, and UDP genuinely 
> supports 64k messages. However, if fragmentation becomes necessary, you need to 
> receive each and every frame in order to reconstruct the whole message. Therefore it 
> is not recommended to send very large messages over udp. RFC3164 even limits the 
> size in 1024 octets, this is not enforced by syslog-ng though.

Right makes perfect sense and seems so obvious now that you said it.

> 
> > I assumed it would also be the case with TCP, even if it is stream based.
> 
> With TCP, the underlying packet boundaries are not available to the application, thus it
> cannot be used to infer the message framing solely from the tcp.
> 
> > 
> > So the potential problems I see are:
> > 1. tcp() doesn't really support no-multi-line (bug) or
> 
> it does, but it does nothing.
> 
> > 2. the docs should be updated to not say it supports no-multi-line (bug) or
> 
> well, this should be clarified there, I agree.
> 
> Algernon has already experimented with support for multiple lines over stream 
> like transport, but it's not yet integrated. It used MIME-style line continuations, 
> e.g. the first character of subsequent must be white space. It's used on the 
> modern /dev/kmsg interface of the linux kernel, but it'll probably work for tcp too.

Ah, that sounds pretty fantastic.

Anyhow, thanks again for the response, seems like only a little documentation would be helpful for those that overlook the finer points of transports with
regards to the no-multi-line flag :)

- Mike


-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.