[syslog-ng] Some logs written, some are not

[Jason Brown]

Thanks.  I’ll try disabling the internal() source.

What’s happening is this:
I have 3 kubernetes pods starting up (I don’t think the issue is related to kubernetes at all.  I think we can treat these pods as 3 distinct servers and applications starting up).
Each pod sends logs to the remote syslog-ng server.  There are some logs sent via UDP, and others via TCP.
The UDP logs always get created.
The TCP logs only get created sometimes.  Sometimes they’ll get created for all 3 pods.  Sometimes for just 1.  Rarely, none get created.
Once a log does get created, the log continues to be written with no problems.

Viewing a packet capture, I see TCP syslog traffic from the pods to the syslog server, even if no file is created.  I’m not seeing any TCP issues overall.

Thanks,
Jason

> On 12. Jun 2020, at 10:11, Fabien Wernli <wernli at in2p3.fr> wrote:
> 
> Hi Jason,
> 
> On Fri, Jun 12, 2020 at 09:58:40AM +0200, Jason Brown wrote:
>> It looks like the cause of the “Error processing log message: <-1>” was indeed the logback configuration.  Changing the priority to something valid cleans up that error.  
>> Unfortunately, it looks like that wasn’t the source of the problem.  Digging in a little deeper, I’m seeing this message:
>> 
>> Jun 12 07:37:03 s_local at syslog syslog-ng[28873]: Syslog connection accepted; fd='19', client='AF_INET(10.13.97.36:49346)', local='AF_INET(0.0.0.0:514)'
>> Jun 12 07:37:03 s_local at syslog syslog-ng[28873]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='Syslog connection accepted; fd=\'19\', client=\'AF_INET(10.13.97.36:49346)\', local=\'AF_INET(0.0.0.0:514)\'', first-suppressed-msg='>>>>>> filter rule evaluation begin; rule=\'f_auth\', location=\'/etc/syslog-ng/syslog-ng.conf:136:32\', msg=\'0x11c1bc0\’’  
>> 
>> So I’m assuming it’s likely a flow control issue.
> 
> I've already seen this message. It's probably unrelated to your issue which
> I forgot what it was about. This message means that the internal() source,
> which is referenced somewhere in your configuration (explicitly or
> implicitly through an scl) which contains messages from syslog-ng itself
> produces a feedback loop in a log path. This snowball effect is detected by
> syslog-ng and action is taken to keep syslog-ng from exploding
> catastrophically ;-)
> 
> You could disable the internal() source, but usually isolating it in its
> separate log path is the way to go.
> 
> Now back to your initial issue, what's happening exactly ?
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>