https://bugzilla.balabit.com/show_bug.cgi?id=215 --- Comment #2 from Mike w <balabit@32ths.com> 2013-01-03 15:51:28 --- Hello Balazs, Thanks for the thorough response and explanations, I responded inline.
(In reply to comment #0)
So after doing some test cases, I've noticed that on no-multi-line is not stripping newlines on v3.3.7. I've not tested previous versions yet.
no-multi-line is a transport independent flag and it is applied to the message as decoded by the underlying transport protocol. The udp transport merely encloses messages into their own UDP frame, embedded newlines are thus possible, and no-multi-line removes them by replacing them with a space.
When using the udp() source, it works just fine.
My test case:
# cat blah line1 line2 line3
# cat blah | nc -r -n myhost 514
And on "myhost" Jan 2 12:25:16 192.2.0.1 line1 Jan 2 12:25:16 192.2.0.1 line2 Jan 2 12:25:16 192.2.0.1 line3
this is the expected behaviour. If you want embedded newlines in messages you need to use a transport that supports those. Such is rfc5424 style messages used by the syslog driver.
Yes this makes sense now.
What do you want to accomplish exactly?
I was hoping to accept windows logs from a snare universal forwarder from multiple devices. It sends multi-line windows logs but can only do it over TCP that is not rfc5424 as far as I can tell and I wanted these messages to be reduced to one line.
So I assume this has something to do with it:
https://bugzilla.balabit.com/show_bug.cgi?id=74
Particularly about newlines being message terminators over tcp(),
exactly.
Maybe there is a way to adjust the terminator if this is the case?
it's not possible right now, I'm afraid.
This is what I was really needing. The terminator in my case would need to be \n\n since this is the true EOM with these logs and syslog-ng could then determine the end of a message in the logs. As it stands, it looks like to get this working on the syslog-ng side I would need to use program(), find true EOM (\n\n), strip the single \n and then return the message as I want it.
I must admit, I was curious what syslog-ng would do if the message size exceeded the mtu of the network. With UDP, I assume the end of the packet is the end of the message so any messages large enough to span two udp packets would actually be two log entries.
This is not true. IP level fragmentation happens below the UDP level, and UDP genuinely supports 64k messages. However, if fragmentation becomes necessary, you need to receive each and every frame in order to reconstruct the whole message. Therefore it is not recommended to send very large messages over udp. RFC3164 even limits the size in 1024 octets, this is not enforced by syslog-ng though.
Right makes perfect sense and seems so obvious now that you said it.
I assumed it would also be the case with TCP, even if it is stream based.
With TCP, the underlying packet boundaries are not available to the application, thus it cannot be used to infer the message framing solely from the tcp.
So the potential problems I see are: 1. tcp() doesn't really support no-multi-line (bug) or
it does, but it does nothing.
2. the docs should be updated to not say it supports no-multi-line (bug) or
well, this should be clarified there, I agree.
Algernon has already experimented with support for multiple lines over stream like transport, but it's not yet integrated. It used MIME-style line continuations, e.g. the first character of subsequent must be white space. It's used on the modern /dev/kmsg interface of the linux kernel, but it'll probably work for tcp too.
Ah, that sounds pretty fantastic. Anyhow, thanks again for the response, seems like only a little documentation would be helpful for those that overlook the finer points of transports with regards to the no-multi-line flag :) - Mike -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.