[syslog-ng] RFC5424 sending - dest=unix-stream. Framing?

[Declan White]

On Fri, Feb 02, 2018 at 07:01:55PM +0000, Declan White wrote:
> Yeah, disregard me. Sorry. 
> Looks like the tunnel I'm using is unclean..

Well, I fixed the tunnel, but it turned out to be a red herring. It seems I've got two different problems in syslog-ng (on Solaris) :

1) If I read from a unix-stream->unix-stream source, recvmsg() reliably EFAULT's the second read of a new connection. Unless I attach a library call tracer.. and then it won't. Heisenbug...
	10109/11:       write(2, 0x00059898, 58)                        = 58
	10109/11:          S e t t i n g   v a l u e ;   m s g = ' 4 7 b 1 0 ' ,   n a m e
	10109/11:          = ' S O U R C E ' ,   v a l u e = ' i n _ t e s t '
	10109/11:       write(2, "\n", 1)                               = 1
	10109/11:       recvmsg(15, 0xFE66B4B4, 0)                      Err#14 EFAULT

I think my build env may have eaten something bad. I'm recompiling all the dependant libs with GCC, and dumping libwrap, to see if that changes anything.

Is any of this compiler noise relevant? :
	../source/syslog-ng-3.12.1/modules/afsocket/socket-options-inet.c:47:8: warning: unused variable 'on' [-Wunused-variable]
	   gint on = 1;
	../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:52:35: warning: format '%d' expects argument of type 'int', but argument 4 has type 'pid_t {aka long int}' [-Wformat=]
	   g_snprintf(buf, buflen, "/proc/%d/%s", pid, proc_file);
	../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:138:1: warning: '_add_nv_pair_proc_readlink' defined but not used [-Wunused-function]
	../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:118:1: warning: '_add_nv_pair_proc_read_argv' defined but not used [-Wunused-function
	../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:104:1: warning: '_add_nv_pair_proc_read_unless_unset' defined but not used [-Wunused-function]
	../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:38:1: warning: '_add_nv_pair_int' defined but not used [-Wunused-function]
	../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c: In function 'log_transport_streams_new':
	../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c:97:14: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
	   self->read = log_transport_streams_read;
	../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c: In function 'afstreams_init_door':
	../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c:133:11: warning: implicit declaration of function 'close'; did you mean 'pclose'? [-Wimplicit-function-declaration]
	           close(fd);

2) If I avoid that unix-stream source and socat the client from a unix domain socket destination into a syslog port source, syslog-ng now has a mismatch of a source sending no framing, and a destination expecting it.

syslog()
	write(5, 0x000721E8, 60)                        = 60
	   5 7   < 1 3 > 1   2 0 1 8 - 0 2 - 0 3 T 0 3 : 1 1 : 2 0 + 0 0 :
	   0 0   o p s t e s t 6 a   -   -   -   -   t e s t 0 8\n

unix-stream()
	read(6, 0x00072B00, 8192)                       = 57
	   < 1 3 > 1   2 0 1 8 - 0 2 - 0 3 T 0 2 : 0 5 : 0 5 + 0 0 : 0 0
	   o p s t e s t 6 a   -   -   -   -   t e s t 0 7\n

I take it that if I did something evil like faking up a length header with length() in a template() I might get unix-stream -> syslog working?


> On Fri, Feb 02, 2018 at 06:48:09AM +0100, Scheidler, Bal?zs wrote:
> > If the new lines at the end are ok, and the messages dont contain newlines
> > themselves, you should be fine. If both destination and source is
> > unix-STREAM(). Don't you hapoen to use unix-dgram() somewhere? That would
> > take the datagram fragments as individual messages.
> > 
> > Can you show me the framing errors you mention?
> > 
> > 
> > On Feb 1, 2018 22:33, "Declan White" <declanw at is.bbc.co.uk> wrote:
> > 
> > I have syslog-ng pumping data into a unix-stream destination
> > socketfilething.
> > 
> > Data going into this domain socket has lots of adventures, and then emerges
> > as a source unix-stream on another host.
> > Both have flags(syslog-protocol), but the data gets framing errors at the
> > receiver.
> > 
> > I think it's become a bytestream (instead of a message stream) enroute, and
> > the recv() is relying on only getting one message at a time, because it
> > can't know the message length any other way, as unix-stream gets messages
> > without framing(?).
> > 
> > Is there a way to salvage this? (I noticed TLS might add framing, but since
> > the tunnel is already TLS ...)