Just to update: I’m getting the same exact behaviour using rsyslog as well as syslog-ng. So I’m going to attribute it to an issue with logback. Thanks, Jason
On 12. Jun 2020, at 10:18, Jason Brown <jbrown@boxconsulting.net> wrote:
Thanks. I’ll try disabling the internal() source.
What’s happening is this: I have 3 kubernetes pods starting up (I don’t think the issue is related to kubernetes at all. I think we can treat these pods as 3 distinct servers and applications starting up). Each pod sends logs to the remote syslog-ng server. There are some logs sent via UDP, and others via TCP. The UDP logs always get created. The TCP logs only get created sometimes. Sometimes they’ll get created for all 3 pods. Sometimes for just 1. Rarely, none get created. Once a log does get created, the log continues to be written with no problems.
Viewing a packet capture, I see TCP syslog traffic from the pods to the syslog server, even if no file is created. I’m not seeing any TCP issues overall.
Thanks, Jason
On 12. Jun 2020, at 10:11, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Jason,
On Fri, Jun 12, 2020 at 09:58:40AM +0200, Jason Brown wrote:
It looks like the cause of the “Error processing log message: <-1>” was indeed the logback configuration. Changing the priority to something valid cleans up that error. Unfortunately, it looks like that wasn’t the source of the problem. Digging in a little deeper, I’m seeing this message:
Jun 12 07:37:03 s_local@syslog syslog-ng[28873]: Syslog connection accepted; fd='19', client='AF_INET(10.13.97.36:49346)', local='AF_INET(0.0.0.0:514)' Jun 12 07:37:03 s_local@syslog syslog-ng[28873]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='Syslog connection accepted; fd=\'19\', client=\'AF_INET(10.13.97.36:49346)\', local=\'AF_INET(0.0.0.0:514)\'', first-suppressed-msg='>>>>>> filter rule evaluation begin; rule=\'f_auth\', location=\'/etc/syslog-ng/syslog-ng.conf:136:32\', msg=\'0x11c1bc0\’’
So I’m assuming it’s likely a flow control issue.
I've already seen this message. It's probably unrelated to your issue which I forgot what it was about. This message means that the internal() source, which is referenced somewhere in your configuration (explicitly or implicitly through an scl) which contains messages from syslog-ng itself produces a feedback loop in a log path. This snowball effect is detected by syslog-ng and action is taken to keep syslog-ng from exploding catastrophically ;-)
You could disable the internal() source, but usually isolating it in its separate log path is the way to go.
Now back to your initial issue, what's happening exactly ?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq