[syslog-ng] multiple file sources, worked - some have now gone silent

Wed Feb 21 16:46:47 UTC 2018

Hello Declan!

Thanks for sharing the details of these issues*.*
Reading your letters I tried to focus on points where we could help. I have
highlighted them below with "|" and answered them the best I could.

*Mainly I see 2 kinds of issues:*
* * Compilation problems on Solaris*

Although we don't support Solaris officially we received and solved compile
related issues on Solaris.
I would recommend to submit these on Github with details to discuss it.

* * The reported destination dropping messages (or "silent sources" as
originally reported ) in syslog-ng.*

The log contains messages like:
> Feb 12 22:28:24.07 host1 syslog-ng[12121]: Destination reliable queue
> full, dropping message; filename='/var/syslog-ng/syslog-ng-00000.rqf',
> queue_len='3929', mem_buf_size='10000', disk_buf_size='2000000',
> persist_name='afsocket_dd_qfile(stream,localhost.afunix:/
> var/syslog-ng/logserver.socket)'
> I don't know why it needs to drop messages when the source is a file and
> the flow-control is on.

The log message is not dropped, this shows that the debug message can
mislead.
Message dropping is handled in a layer above the place where this function
is called.
I think this should be fixed.

> Tried a later syslog-ng version but the tarball was missing 'configure'.
> One of them was missing the bundled json-c.
> Needed an empty "json_object_private.h" in the include path (should be
> another patch, but it was easier just to touch the file).

Can you please specify which tarballs do you mean?

*Some word about your use case:*

> My use case is strangely simple. I want changes to a list of files on one
> host replicated to another host, reliably. Reliably means accounting for
> any network and host disruption, file truncation or rotation.
> This may seem straightforward but there is no such software. People I've
> tracked down in the same situation are just running rsync in while(1)
> loops, which doesn't scale. (Also, I've seen rsync protocol-deadlock on
> big-v-little-endian + 32v64 + differing-raw-directory-order weirdness
> before).

...

If this is all blowing up because the patches I applied to get it to
> compile weren't thread safe, that would be appropriately ironic.

As written your log messages are not necessarily conforming to any syslog
protocols (which would not be a problem itself) and could be very special
too ("... UNTRUSTED application logs in a verbatim-recoverable manner (e.g.
NUL chars, logs with no newlines) ...").
Your use case is quite special: file replication/transfer without any
constraint about the format of the log message while file truncation can
happen.
Please note that syslog-ng is designed to sequentially read from file
sources, if your applications can truncate the file anytime that could lead
(in some cases) to message loss!
In our admin guide we state that after a rotation you must reload/restart
syslog-ng.

Best Regards,
Gabor

On Fri, Feb 16, 2018 at 7:18 PM, Declan White <declanw at is.bbc.co.uk> wrote:

> On Fri, Feb 16, 2018 at 01:28:02PM +0100, Balazs Scheidler wrote:
> > Hi Declan,
> >
> > On Thu, Feb 15, 2018 at 6:13 PM, Declan White <declanw at is.bbc.co.uk>
> wrote:
> >
> > I understand the sentiment and rest assured what you described is not an
> > intended behaviour.
> >
> > Frankly, you are not very helpful here. It is an integral part of open
> > source that users are also contributors and help forming the product
> and/or
> > fix bugs. By asserting that there's a problem on your side, without
> > providing details to help us fix it, and then calling it a deal breaker
> > will not solve the issue.
>
> And I understand that sentiment :) But when I found it had silently broken
> doing something simple, my priority switched from tracing it/helping you
> fix it, to running away screaming. I am in survival mode now.
>
> I spent a month of late nights trying to get rsyslog to work. When it
> vomited thready madness I wisely ran away screaming.
>
> I spent the next month of late nights trying to get syslog-ng working (and
> patching each of the dependancies that themselves didn't compile, and then
> compiling GCC itself when I found you were serious about being GCC-only).
> It then silently broke when the receiver blipped, and complained about
> dropping messages, just shoveling only basic files, in reliable mode, on
> default settings.
>
> Such an obvious breakage in such an obvious usage case almost certainly
> means it doesn't do this on Linux or your other test platforms, and that
> means it will take you a long time to find out what it is about my
> build/env/OSver/threadmodel that is triggering this. And that's assuming
> anyone still cares about Sol10.
>
> That would take me some time working with you to find. I do not have that
> time. I have negative time. I am shedding tears of hysterical laughter and
> coding a new file relay protocol in perl right now, doing daily status
> reports to higherups about the huge delays making a simple reliable file
> relay.
>
> > The premium edition may or may not be for you, but there you could at
> least
> > have some expectations wrt. deal breakers and stuff, as you would be
> paying
> > money in exchange for service and product. And I am not saying that we
> > leave the open source as garbage. We do everything to keep it as stable
> and
> > featureful as possible.
>
> Yes, I dangled that options at the higher ups. I'm desperate for a
> reliable relay protocol (hence the attempt at rsyslog).
> But seeing as I can't guarantee syslog-ng will relay UNTRUSTED application
> logs in a verbatim-recoverable manner (e.g. NUL chars, logs with no
> newlines), I would eventually have to move away from syslog-ng anyway.
>
> The stars are not aligned. The stars are in fact on fire.
>
> > Cheers,
> > --
> > Bazsi
>
> > ____________________________________________________________
> __________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
> --
> Declan White
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180221/d9adf75e/attachment.html>