[syslog-ng] Some logs written, some are not
jbrown at boxconsulting.net
Fri Jun 12 08:18:19 UTC 2020
Thanks. I’ll try disabling the internal() source.
What’s happening is this:
I have 3 kubernetes pods starting up (I don’t think the issue is related to kubernetes at all. I think we can treat these pods as 3 distinct servers and applications starting up).
Each pod sends logs to the remote syslog-ng server. There are some logs sent via UDP, and others via TCP.
The UDP logs always get created.
The TCP logs only get created sometimes. Sometimes they’ll get created for all 3 pods. Sometimes for just 1. Rarely, none get created.
Once a log does get created, the log continues to be written with no problems.
Viewing a packet capture, I see TCP syslog traffic from the pods to the syslog server, even if no file is created. I’m not seeing any TCP issues overall.
> On 12. Jun 2020, at 10:11, Fabien Wernli <wernli at in2p3.fr> wrote:
> Hi Jason,
> On Fri, Jun 12, 2020 at 09:58:40AM +0200, Jason Brown wrote:
>> It looks like the cause of the “Error processing log message: <-1>” was indeed the logback configuration. Changing the priority to something valid cleans up that error.
>> Unfortunately, it looks like that wasn’t the source of the problem. Digging in a little deeper, I’m seeing this message:
>> Jun 12 07:37:03 s_local at syslog syslog-ng: Syslog connection accepted; fd='19', client='AF_INET(10.13.97.36:49346)', local='AF_INET(0.0.0.0:514)'
>> Jun 12 07:37:03 s_local at syslog syslog-ng: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='Syslog connection accepted; fd=\'19\', client=\'AF_INET(10.13.97.36:49346)\', local=\'AF_INET(0.0.0.0:514)\'', first-suppressed-msg='>>>>>> filter rule evaluation begin; rule=\'f_auth\', location=\'/etc/syslog-ng/syslog-ng.conf:136:32\', msg=\'0x11c1bc0\’’
>> So I’m assuming it’s likely a flow control issue.
> I've already seen this message. It's probably unrelated to your issue which
> I forgot what it was about. This message means that the internal() source,
> which is referenced somewhere in your configuration (explicitly or
> implicitly through an scl) which contains messages from syslog-ng itself
> produces a feedback loop in a log path. This snowball effect is detected by
> syslog-ng and action is taken to keep syslog-ng from exploding
> catastrophically ;-)
> You could disable the internal() source, but usually isolating it in its
> separate log path is the way to go.
> Now back to your initial issue, what's happening exactly ?
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
More information about the syslog-ng