[syslog-ng] RFC5424 sending - dest=unix-stream. Framing?

[Declan White]

On Sat, Feb 03, 2018 at 03:23:06PM +0100, Scheidler, Bal?zs wrote:
> You could use tcp(flags(syslog-protocol)) to send rfc5424 without framing.

I was already sending without framing but, now you've mentioned it, I've gone over the differences between network(flags(syslog-protocol)) and syslog() again.
I'd thought they were synonymous/equivalent - I was mistaken!

EUREKA! Switching the receiver from syslog() to network(flags(syslog-protocol)) gets everything working.

Thanks! :)

- Declan

PS: the recompile didn't kill the Solaris heisenbug, but I'll happily take this escape route instead.
 
> On Feb 3, 2018 06:16, "Declan White" <declanw at is.bbc.co.uk> wrote:
> 
> > On Fri, Feb 02, 2018 at 07:01:55PM +0000, Declan White wrote:
> > > Yeah, disregard me. Sorry.
> > > Looks like the tunnel I'm using is unclean..
> >
> > Well, I fixed the tunnel, but it turned out to be a red herring. It seems
> > I've got two different problems in syslog-ng (on Solaris) :
> >
> > 1) If I read from a unix-stream->unix-stream source, recvmsg() reliably
> > EFAULT's the second read of a new connection. Unless I attach a library
> > call tracer.. and then it won't. Heisenbug...
> >         10109/11:       write(2, 0x00059898, 58)                        =
> > 58
> >         10109/11:          S e t t i n g   v a l u e ;   m s g = ' 4 7 b 1
> > 0 ' ,   n a m e
> >         10109/11:          = ' S O U R C E ' ,   v a l u e = ' i n _ t e s
> > t '
> >         10109/11:       write(2, "\n", 1)                               = 1
> >         10109/11:       recvmsg(15, 0xFE66B4B4, 0)
> > Err#14 EFAULT
> >
> > I think my build env may have eaten something bad. I'm recompiling all the
> > dependant libs with GCC, and dumping libwrap, to see if that changes
> > anything.
> >
> > Is any of this compiler noise relevant? :
> >         ../source/syslog-ng-3.12.1/modules/afsocket/socket-options-inet.c:47:8:
> > warning: unused variable 'on' [-Wunused-variable]
> >            gint on = 1;
> >         ../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:52:35:
> > warning: format '%d' expects argument of type 'int', but argument 4 has
> > type 'pid_t {aka long int}' [-Wformat=]
> >            g_snprintf(buf, buflen, "/proc/%d/%s", pid, proc_file);
> >         ../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:138:1:
> > warning: '_add_nv_pair_proc_readlink' defined but not used
> > [-Wunused-function]
> >         ../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:118:1:
> > warning: '_add_nv_pair_proc_read_argv' defined but not used
> > [-Wunused-function
> >         ../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:104:1:
> > warning: '_add_nv_pair_proc_read_unless_unset' defined but not used
> > [-Wunused-function]
> >         ../source/syslog-ng-3.12.1/modules/afsocket/transport-unix-socket.c:38:1:
> > warning: '_add_nv_pair_int' defined but not used [-Wunused-function]
> >         ../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c: In
> > function 'log_transport_streams_new':
> >         ../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c:97:14:
> > warning: assignment from incompatible pointer type
> > [-Wincompatible-pointer-types]
> >            self->read = log_transport_streams_read;
> >         ../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c: In
> > function 'afstreams_init_door':
> >         ../source/syslog-ng-3.12.1/modules/afstreams/afstreams.c:133:11:
> > warning: implicit declaration of function 'close'; did you mean 'pclose'?
> > [-Wimplicit-function-declaration]
> >                    close(fd);
> >
> > 2) If I avoid that unix-stream source and socat the client from a unix
> > domain socket destination into a syslog port source, syslog-ng now has a
> > mismatch of a source sending no framing, and a destination expecting it.
> >
> > syslog()
> >         write(5, 0x000721E8, 60)                        = 60
> >            5 7   < 1 3 > 1   2 0 1 8 - 0 2 - 0 3 T 0 3 : 1 1 : 2 0 + 0 0 :
> >            0 0   o p s t e s t 6 a   -   -   -   -   t e s t 0 8\n
> >
> > unix-stream()
> >         read(6, 0x00072B00, 8192)                       = 57
> >            < 1 3 > 1   2 0 1 8 - 0 2 - 0 3 T 0 2 : 0 5 : 0 5 + 0 0 : 0 0
> >            o p s t e s t 6 a   -   -   -   -   t e s t 0 7\n
> >
> > I take it that if I did something evil like faking up a length header with
> > length() in a template() I might get unix-stream -> syslog working?
> >
> >
> > > On Fri, Feb 02, 2018 at 06:48:09AM +0100, Scheidler, Bal?zs wrote:
> > > > If the new lines at the end are ok, and the messages dont contain
> > newlines
> > > > themselves, you should be fine. If both destination and source is
> > > > unix-STREAM(). Don't you hapoen to use unix-dgram() somewhere? That
> > would
> > > > take the datagram fragments as individual messages.
> > > >
> > > > Can you show me the framing errors you mention?
> > > >
> > > >
> > > > On Feb 1, 2018 22:33, "Declan White" <declanw at is.bbc.co.uk> wrote:
> > > >
> > > > I have syslog-ng pumping data into a unix-stream destination
> > > > socketfilething.
> > > >
> > > > Data going into this domain socket has lots of adventures, and then
> > emerges
> > > > as a source unix-stream on another host.
> > > > Both have flags(syslog-protocol), but the data gets framing errors at
> > the
> > > > receiver.
> > > >
> > > > I think it's become a bytestream (instead of a message stream)
> > enroute, and
> > > > the recv() is relying on only getting one message at a time, because it
> > > > can't know the message length any other way, as unix-stream gets
> > messages
> > > > without framing(?).
> > > >
> > > > Is there a way to salvage this? (I noticed TLS might add framing, but
> > since
> > > > the tunnel is already TLS ...)

-- 
Declan White