https://bugzilla.balabit.com/show_bug.cgi?id=215 --- Comment #1 from Balazs Scheidler <bazsi@balabit.hu> 2013-01-02 20:19:46 --- (In reply to comment #0)

So after doing some test cases, I've noticed that on no-multi-line is not stripping newlines on v3.3.7. I've not tested previous versions yet.

Here is my syslog-ng config:

source s_net_tcp { tcp(ip(0.0.0.0) port(514) flags(no-multi-line)); };

destination all { file("/var/log/all"); };

log { source(s_net_tcp); destination(all); flags(final); };

I've also attempted to do this with no-parse flags passed in to no avail.

no-multi-line is a transport independent flag and it is applied to the message as decoded by the underlying transport protocol. The udp transport merely encloses messages into their own UDP frame, embedded newlines are thus possible, and no-multi-line removes them by replacing them with a space.

When using the udp() source, it works just fine.

My test case:

# cat blah line1 line2 line3

# cat blah | nc -r -n myhost 514

And on "myhost" Jan 2 12:25:16 192.2.0.1 line1 Jan 2 12:25:16 192.2.0.1 line2 Jan 2 12:25:16 192.2.0.1 line3

this is the expected behaviour. If you want embedded newlines in messages you need to use a transport that supports those. Such is rfc5424 style messages used by the syslog driver. What do you want to accomplish exactly?

So I assume this has something to do with it:

https://bugzilla.balabit.com/show_bug.cgi?id=74

Particularly about newlines being message terminators over tcp(),

exactly.

however, why do the docs even say tcp() supports the no-multi-line flag?

yup, it supports it, but it fails to find an actual newline there. but the same functionality is used when regardless of the transport.

Maybe there is a way to adjust the terminator if this is the case?

it's not possible right now, I'm afraid.

I must admit, I was curious what syslog-ng would do if the message size exceeded the mtu of the network. With UDP, I assume the end of the packet is the end of the message so any messages large enough to span two udp packets would actually be two log entries.

This is not true. IP level fragmentation happens below the UDP level, and UDP genuinely supports 64k messages. However, if fragmentation becomes necessary, you need to receive each and every frame in order to reconstruct the whole message. Therefore it is not recommended to send very large messages over udp. RFC3164 even limits the size in 1024 octets, this is not enforced by syslog-ng though.

I assumed it would also be the case with TCP, even if it is stream based.

With TCP, the underlying packet boundaries are not available to the application, thus it cannot be used to infer the message framing solely from the tcp.

So the potential problems I see are: 1. tcp() doesn't really support no-multi-line (bug) or

it does, but it does nothing.

2. the docs should be updated to not say it supports no-multi-line (bug) or

well, this should be clarified there, I agree.

3. I've overlooked the doc references or 4. I'm missing some option (terminator?) in order to get tcp() to work with no-multi-line. (env)

Algernon has already experimented with support for multiple lines over stream like transport, but it's not yet integrated. It used MIME-style line continuations, e.g. the first character of subsequent must be white space. It's used on the modern /dev/kmsg interface of the linux kernel, but it'll probably work for tcp too. What I couldn't decipher exactly, what you'd like to accomplish. Can you elaborate on that? Thanks. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

https://bugzilla.balabit.com/show_bug.cgi?id=215 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon@balabit.hu AssignedTo|bazsi@balabit.hu |algernon@balabit.hu --- Comment #3 from Gergely Nagy <algernon@balabit.hu> 2013-01-03 16:58:15 --- (In reply to comment #2)

...

What do you want to accomplish exactly?

I was hoping to accept windows logs from a snare universal forwarder from multiple devices. It sends multi-line windows logs but can only do it over TCP that is not rfc5424 as far as I can tell and I wanted these messages to be reduced to one line.

This will be possible once the multiline work is merged, which will happen early in the 3.5 development cycle. [...]

This is what I was really needing. The terminator in my case would need to be \n\n since this is the true EOM with these logs and syslog-ng could then determine the end of a message in the logs. As it stands, it looks like to get this working on the syslog-ng side I would need to use program(), find true EOM (\n\n), strip the single \n and then return the message as I want it.

Yep, this is the most straightforward workaround right now.

...

Algernon has already experimented with support for multiple lines over stream like transport, but it's not yet integrated. It used MIME-style line continuations, e.g. the first character of subsequent must be white space. It's used on the modern /dev/kmsg interface of the linux kernel, but it'll probably work for tcp too.

Ah, that sounds pretty fantastic.

My multiline work supports mime-style line continuations already (but that won't help with windows logs), and work is in progress to extend it further, so it supports windows logs and custom EOM marks too. So early 3.5 will have everything in place that you need. If I could be so bold, I'd like to ask you to test 3.5, once these features are in, so that we can make multiline support cover as many use cases as possible. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

https://bugzilla.balabit.com/show_bug.cgi?id=215 --- Comment #4 from Mike w <balabit@32ths.com> 2013-01-03 19:11:50 ---

My multiline work supports mime-style line continuations already (but that won't help with windows logs), and work is in progress to extend it further, so it supports windows logs and custom EOM marks too.

So early 3.5 will have everything in place that you need. If I could be so bold, I'd like to ask you to test 3.5, once these features are in, so that we can make multiline support cover as many use cases as possible.

Sure, I'd be happy to test. - Mike -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.